Discourse app stuck in Not Responding state after full Cloudron restore (dry run)

[jdaviescoates]

I'm in the process of migrating my Cloudron to a new server.

I've done a dry run full restore on the new server. All apps and services are up and running apart from a Discourse app. It doesn't have any plugins or anything enabled to disable, so not sure how best to proceed.

This is what I'm seeing in the logs:

box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
Sep 01 00:35:47box:tasks update 13828: {"percent":100,"message":"Done"}
Sep 01 00:35:48box:taskworker Task took 1.276 seconds
Sep 01 00:35:48box:tasks setCompleted - 13828: {"result":null,"error":null}
Sep 01 00:35:48box:tasks update 13828: {"percent":100,"result":null,"error":null}
Sep 01 00:35:47==> Configuring discourse
Sep 01 00:35:47==> Creating symlinks for built-in plugins
Sep 01 00:35:47==> Changing permissions
Sep 01 00:35:47==> Migrating database
Sep 01 00:35:482023-08-31 23:35:48,345 INFO spawned: 'redis' with pid 12
Sep 01 00:35:482023-08-31 23:35:48,346 INFO spawned: 'redis-service' with pid 13
Sep 01 00:35:4812:C 31 Aug 2023 23:35:48.350 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
Sep 01 00:35:4812:C 31 Aug 2023 23:35:48.350 # Redis version=7.0.11, bits=64, commit=00000000, modified=0, pid=12, just started
Sep 01 00:35:4812:C 31 Aug 2023 23:35:48.350 # Configuration loaded
Sep 01 00:35:4812:M 31 Aug 2023 23:35:48.350 * monotonic clock: POSIX clock_gettime
Sep 01 00:35:4812:M 31 Aug 2023 23:35:48.351 * Running mode=standalone, port=6379.
Sep 01 00:35:4812:M 31 Aug 2023 23:35:48.351 # Server initialized
Sep 01 00:35:4812:M 31 Aug 2023 23:35:48.351 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
Sep 01 00:35:4812:M 31 Aug 2023 23:35:48.351 * Loading RDB produced by version 7.0.11
Sep 01 00:35:4812:M 31 Aug 2023 23:35:48.351 * RDB age 10 seconds
Sep 01 00:35:4812:M 31 Aug 2023 23:35:48.351 * RDB memory usage when created 2.34 Mb
Sep 01 00:35:4812:M 31 Aug 2023 23:35:48.354 * Done loading RDB, keys loaded: 2563, keys expired: 0.
Sep 01 00:35:4812:M 31 Aug 2023 23:35:48.354 * DB loaded from disk: 0.003 seconds
Sep 01 00:35:4812:M 31 Aug 2023 23:35:48.354 * Ready to accept connections
Sep 01 00:35:48Redis service endpoint listening on http://:::3000
Sep 01 00:35:48fatal: not a git repository (or any of the parent directories): .git
Sep 01 00:35:492023-08-31 23:35:49,438 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Sep 01 00:35:492023-08-31 23:35:49,438 INFO success: redis-service entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Sep 01 00:35:50=> Healtheck error: Error: connect ECONNREFUSED 172.18.16.86:80
Sep 01 00:35:50==> Pre-compiling assets
Sep 01 00:35:51fatal: not a git repository (or any of the parent directories): .git
Sep 01 00:35:52yarn run v1.22.19
Sep 01 00:35:52$ /app/code/app/assets/javascripts/node_modules/.bin/ember build -prod
Sep 01 00:35:52warning Cannot find a suitable global folder. Tried these: "/usr/local, /home/cloudron/.yarn"
Sep 01 00:35:53⠋ BuildingEnvironment: production
Sep 01 00:36:00=> Healtheck error: Error: connect ECONNREFUSED 172.18.16.86:80
Sep 01 00:36:10=> Healtheck error: Error: connect ECONNREFUSED 172.18.16.86:80
Sep 01 00:36:17⠋ Building⠙ building... ⠹ building... [Babel: @ember/test-helpers > applyPatches]⠸ building... [Babel: ember-exam > applyPatches]⠼ building... [Babel: ember-source > applyPatches]⠴ building... [ember.js]⠦ building... [Babel: @ember/legacy-built-in-components > applyPatches]⠧ building... [Babel: ember-resize-observer-service > applyPatches]⠇ building... [ember-auto-import-analyzer]⠏ building... [ConfigLoader]⠋ building... [TemplateCompiler > applyPatches]⠙ building... [Babel: discourse > applyPatches]⠹ building... [Babel: discourse > applyPatches]⠸ building... [Babel: discourse > applyPatches]⠼ building... [Babel: discourse > applyPatches]⠴ building... [Babel: discourse > applyPatches]⠦ building... [ember-auto-import-analyzer]⠧ building... [Packaged Application Javascript]⠇ building... [Package /assets/vendor.js]⠏ building... [Babel: discourse > applyPatches]⠋ building... [Babel: discourse > applyPatches]⠙ building... [Concat: Core Tests]⠹ building... [Concat: Test Support JS]⠸ building... [SourceMapConcat]⠼ building... [TemplateCompiler > applyPatches]⠴ building... [Babel: admin > applyPatches]⠦ building... [ember-auto-import-analyzer]⠧ building... [ember-auto-import-analyzer]⠇ building... [SourceMapConcat]⠏ building... [TemplateCompiler > applyPatches]⠋ building... [Babel: discourse-plugins > applyPatches]⠙ building... [SourceMapConcat]⠹ building... [Babel: discourse-plugins > applyPatches]⠸ building... [TemplateCompiler > applyPatches]⠼ building... [Babel: discourse-plugins > applyPatches]⠴ building... [ember-auto-import-webpack]⠦ building... [ember-auto-import-webpack]⠧ building... [ember-auto-import-webpack]⠇ building... [ember-auto-import-webpack]⠏ building... [ember-auto-import-webpack]⠋ building... [ember-auto-import-webpack]⠙ building... [ember-auto-import-webpack]⠹ building... [ember-auto-import-webpack]⠸ building... [ember-auto-import-webpack]⠼ building... [ember-auto-import-webpack]⠴ building... [ember-auto-import-webpack]⠦ building... [ember-auto-import-webpack]⠧ building... [ember-auto-import-webpack]⠇ building... [ember-auto-import-webpack]⠏ building... [ember-auto-import-webpack]⠋ building... [ember-auto-import-webpack]⠙ building... [ember-auto-import-webpack]⠹ building... [ember-auto-import-webpack]⠸ building... [ember-auto-import-webpack]⠼ building... [ember-auto-import-webpack]⠴ building... [ember-auto-import-webpack]⠦ building... [ember-auto-import-webpack]⠧ building... [ember-auto-import-webpack]⠇ building... [ember-auto-import-webpack]⠏ building... [ember-auto-import-webpack]⠋ building... [ember-auto-import-webpack]⠙ building... [ember-auto-import-webpack]⠹ building... [ember-auto-import-webpack]⠸ building... [ember-auto-import-webpack]⠼ building... [ember-auto-import-webpack]⠴ building... [ember-auto-import-webpack]⠦ building... [ember-auto-import-webpack]⠧ building... [ember-auto-import-webpack]⠇ building... [ember-auto-import-webpack]⠏ building... [ember-auto-import-webpack]⠋ building... [ember-auto-import-webpack]⠙ building... [ember-auto-import-webpack]⠹ building... [ember-auto-import-webpack]⠸ building... [ember-auto-import-webpack]⠼ building... [ember-auto-import-webpack]⠴ building... [ember-auto-import-webpack]⠦ building... [ember-auto-import-webpack]⠧ building... [ember-auto-import-webpack]⠇ building... [ember-auto-import-webpack]⠏ building... [ember-auto-import-webpack]⠋ building... [ember-auto-import-webpack]⠙ building... [ember-auto-import-webpack]⠹ building... [ember-auto-import-webpack]⠸ building... [ember-auto-import-webpack]⠼ building... [ember-auto-import-webpack]⠴ building... [ember-auto-import-webpack]⠦ building... [ember-auto-import-webpack]⠧ building... [ember-auto-import-webpack]⠇ building... [ember-auto-import-webpack]⠏ building... [ember-auto-import-webpack]⠋ building... [ember-auto-import-webpack]⠙ building... [ember-auto-import-webpack]⠹ building... [ember-auto-import-webpack]⠸ building... [ember-auto-import-webpack]⠼ building... [ember-auto-import-webpack]⠴ building... [ember-auto-import-webpack]⠦ building... [ember-auto-import-webpack]⠧ building... [TerserWriter]⠇ building... [TerserWriter]⠏ building... [TerserWriter]⠋ building... [TerserWriter]⠙ building... [TerserWriter]⠹ building... [TerserWriter]⠸ building... [TerserWriter]⠼ building... [TerserWriter]⠴ building... [TerserWriter]⠦ building... [TerserWriter]⠧ building... [TerserWriter]⠇ building... [TerserWriter]⠏ building... [TerserWriter]⠋ building... [TerserWriter]⠙ building... [TerserWriter]⠹ building... [TerserWriter]⠸ building... [TerserWriter]⠼ building... [TerserWriter]⠴ building... [TerserWriter]⠦ building... [TerserWriter]⠧ building... [TerserWriter]⠇ building... [TerserWriter]⠏ building... [TerserWriter]⠋ building... [TerserWriter]⠙ building... [TerserWriter]⠹ building... [TerserWriter]⠸ building... [TerserWriter]⠼ building... [TerserWriter]⠴ building... [TerserWriter]⠦ building... [TerserWriter]⠧ building... [TerserWriter]⠇ building... [TerserWriter]⠏ building... [TerserWriter]⠋ building... [TerserWriter]⠙ building... [TerserWriter]⠹ building... [TerserWriter]⠸ building... [TerserWriter]⠼ building... [TerserWriter]⠴ building... [TerserWriter]⠦ building... [TerserWriter]⠧ building... [TerserWriter]⠇ building... [TerserWriter]⠏ building... [TerserWriter]⠋ building... [TerserWriter]⠙ building... [TerserWriter]⠹ building... [TerserWriter]⠸ building... [TerserWriter]⠼ building... [TerserWriter]⠴ building... [TerserWriter]⠦ building... [TerserWriter]⠧ building... [TerserWriter]⠇ building... [TerserWriter]⠏ building... [TerserWriter]⠋ building... [TerserWriter]⠙ building... [TerserWriter]⠹ building... [TerserWriter]⠸ building... [TerserWriter]⠼ building... [TerserWriter]⠴ building... [TerserWriter]⠦ building... [TerserWriter]⠧ building... [TerserWriter]⠇ building... [TerserWriter]⠏ building... [TerserWriter]⠋ building... [TerserWriter]⠙ building... [TerserWriter]⠹ building... [TerserWriter]⠸ building... [TerserWriter]⠼ building... [TerserWriter]⠴ building... [TerserWriter]⠦ building... [TerserWriter]⠧ building... [TerserWriter]⠇ building... [TerserWriter]⠏ building... [TerserWriter]⠋ building... [TerserWriter]⠙ building... [TerserWriter]⠹ building... [TerserWriter]⠸ building... [TerserWriter]⠼ building... [TerserWriter]⠴ building... [TerserWriter]⠦ building... [TerserWriter]⠧ building... [TerserWriter]⠇ building... [TerserWriter]⠏ building... [TerserWriter]⠋ building... [TerserWriter]⠙ building... [TerserWriter]⠹ building... [TerserWriter]⠸ building... [TerserWriter]⠼ building... [TerserWriter]⠴ building... [TerserWriter]⠦ building... [TerserWriter]⠧ building... [TerserWriter]⠇ building... [TerserWriter]⠏ building... [TerserWriter]⠋ building... [TerserWriter]⠙ building... [TerserWriter]⠹ building... [TerserWriter]⠸ building... [TerserWriter]⠼ building... [TerserWriter]⠴ building... [TerserWriter]⠦ building... [TerserWriter]⠧ building... [TerserWriter]⠇ building... [TerserWriter]⠏ building... [TerserWriter]⠋ building... [TerserWriter]Killed
Sep 01 00:36:17error Command failed with exit code 137.
Sep 01 00:36:17info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Sep 01 00:36:18==> Creating directories
Sep 01 00:36:18==> Configuring discourse
Sep 01 00:36:18==> Creating symlinks for built-in plugins
Sep 01 00:36:18==> Changing permissions
Sep 01 00:36:18==> Migrating database
Sep 01 00:36:19fatal: not a git repository (or any of the parent directories): .git
Sep 01 00:36:20=> Healtheck error: Error: connect ECONNREFUSED 172.18.16.86:80
Sep 01 00:36:21==> Pre-compiling assets
Sep 01 00:36:22fatal: not a git repitory (or any of the parent directories): .git
Sep 01 00:36:22yarn run v1.22.

[girish]

@jdaviescoates so nothing happens after that build line in the logs? Usually asset building is a heavy CPU operation, so maybe it's just taking time?

[jdaviescoates]

@girish said in Discourse app stuck in Not Responding state after full Cloudron restore (dry run):

@jdaviescoates so nothing happens after that build line in the logs?

No, nothing happens. Well, it just keeps trying to do the same thing over and over again but failing. I originally tried to post more of the logs but the forum wouldn't allow such a long post.

@girish said in Discourse app stuck in Not Responding state after full Cloudron restore (dry run):

Usually asset building is a heavy CPU operation, so maybe it's just taking time?

This is on a dedicated server with plenty of spare CPU. I does use a lot, but it doesn't achieve anything. I've left it all night, still exactly the same.

The only thing that's changed overnight is that now all my instances of WordPress are in an error state too! (this is all on the dry run restore so no huge deal other than it's slowing down the migrate and at this rate I'll need to do another back-up of the current server and start again...)

The Discourse did seemingly keep running out of memory though, so perhaps that was it. I've just whacked it up and am trying again...

Oh, but now I seem to be running out of disk space too

But that's odd because here is the current server showing I'm using 317.92 GB:

But here is the new server saying I'm somehow now using 429.56 GB!

Somehow the full restore on the new server is taking up over 100GB more space - how is that possible?!?

Looks like the culprit is the "Everything else" which has somehow massively grown by nearly 130GB! WTF?!?

Current 27.87 GB:

New 156.94 GB:

Perhaps something to do with a failed partial backup the dry run tried to do overnight?

Seeing this error message:

Error piping http://172.18.30.2:3000/databases/db737de757d2b64357ad482b3b33ce982d/backup?access_token=b13348ff73804eb6301b8a966fca439b94ea1eee9cbe3cfbcc6b67b9cd4baa3299c7465706acf44e9897d5cb6534badf50ecb7eeb311f1a729cf25743f526323e9cda42f982db27018d16ee7dc145734933e4082713cca5374c0668b1ce8cc1ff136af5adbfa0db608707087b616fb82b1cba0445cff453adf5ea2dc42e4f083 to /home/yellowtent/appsdata/737de757-d2b6-4357-ad48-2b3b33ce982d/postgresqldump: ENOSPC: no space left on device, write

Edit: Oh! But while I was noticing all that, the instance of Discourse did finally manage to start after I gave it 6.5GB RAM!

So now just need to work out why all the WordPress app have crashed and why "Everything else" is so massive...

Ah, it's because of the disk space, seeing a load of these:

Docker Error: Not enough disk space to pull docker image

[jdaviescoates]

OK, so now everything is running again, I've just got this disk space issue...

How can I work out why "Everything" else is taking up nearly 130 GB more space that it was before?

[jdaviescoates]

Seems the Disk Usage graph is wrong, trying to refresh it but it's taking an age...

The little refresh button has been turning for yonks now, nothing seems to be happening.

But if I login via SSH and do df -h I see this:

root@Ubuntu-2204-jammy-amd64-base ~ # df -h
Filesystem                           Size  Used Avail Use% Mounted on
tmpfs                                6.3G  6.5M  6.3G   1% /run
/dev/md2                             436G  271G  143G  66% /

So seemingly there is actually plenty of space left of the drive (I just did a docker prune -a - although it told me that free up 0B - and Cleaned Up backups, so maybe that did it?)

Hard to tell if anything has changed without the Disk Usage graph refreshing...

[jdaviescoates]

How can I force the Disk Usage graphs to refresh? I clicked refresh like half an hour ago or something and nothing has happened, it's just going around and around...

[jdaviescoates]

@jdaviescoates said in Discourse app stuck in Not Responding state after full Cloudron restore (dry run):

How can I force the Disk Usage graphs to refresh? I clicked refresh like half an hour ago or something and nothing has happened, it's just going around and around...

It finally did it and now is showing this:

I wonder what caused Everything else to balloon so much previously?!?

[girish]

@jdaviescoates phew, glad everything is back. Not sure what caused things to balloon up so much though.

[jdaviescoates]

@girish said in Discourse app stuck in Not Responding state after full Cloudron restore (dry run):

Not sure what caused things to balloon up so much though.

Yeah, that's a bit of a mystery