Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse

Cloudron Forum

Apps | Demo | Docs | Install

Discourse app stuck in Not Responding state after full Cloudron restore (dry run)

Scheduled Pinned Locked Moved Discourse
9 Posts 2 Posters 67 Views
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • jdaviescoatesJ Offline
    jdaviescoatesJ Offline
    jdaviescoates
    wrote on last edited by
    #1

    I'm in the process of migrating my Cloudron to a new server.

    I've done a dry run full restore on the new server. All apps and services are up and running apart from a Discourse app. It doesn't have any plugins or anything enabled to disable, so not sure how best to proceed.

    This is what I'm seeing in the logs:

    box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
    Sep 01 00:35:47box:tasks update 13828: {"percent":100,"message":"Done"}
    Sep 01 00:35:48box:taskworker Task took 1.276 seconds
    Sep 01 00:35:48box:tasks setCompleted - 13828: {"result":null,"error":null}
    Sep 01 00:35:48box:tasks update 13828: {"percent":100,"result":null,"error":null}
    Sep 01 00:35:47==> Configuring discourse
    Sep 01 00:35:47==> Creating symlinks for built-in plugins
    Sep 01 00:35:47==> Changing permissions
    Sep 01 00:35:47==> Migrating database
    Sep 01 00:35:482023-08-31 23:35:48,345 INFO spawned: 'redis' with pid 12
    Sep 01 00:35:482023-08-31 23:35:48,346 INFO spawned: 'redis-service' with pid 13
    Sep 01 00:35:4812:C 31 Aug 2023 23:35:48.350 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
    Sep 01 00:35:4812:C 31 Aug 2023 23:35:48.350 # Redis version=7.0.11, bits=64, commit=00000000, modified=0, pid=12, just started
    Sep 01 00:35:4812:C 31 Aug 2023 23:35:48.350 # Configuration loaded
    Sep 01 00:35:4812:M 31 Aug 2023 23:35:48.350 * monotonic clock: POSIX clock_gettime
    Sep 01 00:35:4812:M 31 Aug 2023 23:35:48.351 * Running mode=standalone, port=6379.
    Sep 01 00:35:4812:M 31 Aug 2023 23:35:48.351 # Server initialized
    Sep 01 00:35:4812:M 31 Aug 2023 23:35:48.351 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
    Sep 01 00:35:4812:M 31 Aug 2023 23:35:48.351 * Loading RDB produced by version 7.0.11
    Sep 01 00:35:4812:M 31 Aug 2023 23:35:48.351 * RDB age 10 seconds
    Sep 01 00:35:4812:M 31 Aug 2023 23:35:48.351 * RDB memory usage when created 2.34 Mb
    Sep 01 00:35:4812:M 31 Aug 2023 23:35:48.354 * Done loading RDB, keys loaded: 2563, keys expired: 0.
    Sep 01 00:35:4812:M 31 Aug 2023 23:35:48.354 * DB loaded from disk: 0.003 seconds
    Sep 01 00:35:4812:M 31 Aug 2023 23:35:48.354 * Ready to accept connections
    Sep 01 00:35:48Redis service endpoint listening on http://:::3000
    Sep 01 00:35:48fatal: not a git repository (or any of the parent directories): .git
    Sep 01 00:35:492023-08-31 23:35:49,438 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
    Sep 01 00:35:492023-08-31 23:35:49,438 INFO success: redis-service entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
    Sep 01 00:35:50=> Healtheck error: Error: connect ECONNREFUSED 172.18.16.86:80
    Sep 01 00:35:50==> Pre-compiling assets
    Sep 01 00:35:51fatal: not a git repository (or any of the parent directories): .git
    Sep 01 00:35:52yarn run v1.22.19
    Sep 01 00:35:52$ /app/code/app/assets/javascripts/node_modules/.bin/ember build -prod
    Sep 01 00:35:52warning Cannot find a suitable global folder. Tried these: "/usr/local, /home/cloudron/.yarn"
    Sep 01 00:35:53⠋ BuildingEnvironment: production
    Sep 01 00:36:00=> Healtheck error: Error: connect ECONNREFUSED 172.18.16.86:80
    Sep 01 00:36:10=> Healtheck error: Error: connect ECONNREFUSED 172.18.16.86:80
    Sep 01 00:36:17⠋ Building⠙ building... ⠹ building... [Babel: @ember/test-helpers > applyPatches]⠸ building... [Babel: ember-exam > applyPatches]⠼ building... [Babel: ember-source > applyPatches]⠴ building... [ember.js]⠦ building... [Babel: @ember/legacy-built-in-components > applyPatches]⠧ building... [Babel: ember-resize-observer-service > applyPatches]⠇ building... [ember-auto-import-analyzer]⠏ building... [ConfigLoader]⠋ building... [TemplateCompiler > applyPatches]⠙ building... [Babel: discourse > applyPatches]⠹ building... [Babel: discourse > applyPatches]⠸ building... [Babel: discourse > applyPatches]⠼ building... [Babel: discourse > applyPatches]⠴ building... [Babel: discourse > applyPatches]⠦ building... [ember-auto-import-analyzer]⠧ building... [Packaged Application Javascript]⠇ building... [Package /assets/vendor.js]⠏ building... [Babel: discourse > applyPatches]⠋ building... [Babel: discourse > applyPatches]⠙ building... [Concat: Core Tests]⠹ building... [Concat: Test Support JS]⠸ building... [SourceMapConcat]⠼ building... [TemplateCompiler > applyPatches]⠴ building... [Babel: admin > applyPatches]⠦ building... [ember-auto-import-analyzer]⠧ building... [ember-auto-import-analyzer]⠇ building... [SourceMapConcat]⠏ building... [TemplateCompiler > applyPatches]⠋ building... [Babel: discourse-plugins > applyPatches]⠙ building... [SourceMapConcat]⠹ building... [Babel: discourse-plugins > applyPatches]⠸ building... [TemplateCompiler > applyPatches]⠼ building... [Babel: discourse-plugins > applyPatches]⠴ building... [ember-auto-import-webpack]⠦ building... [ember-auto-import-webpack]⠧ building... [ember-auto-import-webpack]⠇ building... [ember-auto-import-webpack]⠏ building... [ember-auto-import-webpack]⠋ building... [ember-auto-import-webpack]⠙ building... [ember-auto-import-webpack]⠹ building... [ember-auto-import-webpack]⠸ building... [ember-auto-import-webpack]⠼ building... [ember-auto-import-webpack]⠴ building... [ember-auto-import-webpack]⠦ building... [ember-auto-import-webpack]⠧ building... [ember-auto-import-webpack]⠇ building... [ember-auto-import-webpack]⠏ building... [ember-auto-import-webpack]⠋ building... [ember-auto-import-webpack]⠙ building... [ember-auto-import-webpack]⠹ building... [ember-auto-import-webpack]⠸ building... [ember-auto-import-webpack]⠼ building... [ember-auto-import-webpack]⠴ building... [ember-auto-import-webpack]⠦ building... [ember-auto-import-webpack]⠧ building... [ember-auto-import-webpack]⠇ building... [ember-auto-import-webpack]⠏ building... [ember-auto-import-webpack]⠋ building... [ember-auto-import-webpack]⠙ building... [ember-auto-import-webpack]⠹ building... [ember-auto-import-webpack]⠸ building... [ember-auto-import-webpack]⠼ building... [ember-auto-import-webpack]⠴ building... [ember-auto-import-webpack]⠦ building... [ember-auto-import-webpack]⠧ building... [ember-auto-import-webpack]⠇ building... [ember-auto-import-webpack]⠏ building... [ember-auto-import-webpack]⠋ building... [ember-auto-import-webpack]⠙ building... [ember-auto-import-webpack]⠹ building... [ember-auto-import-webpack]⠸ building... [ember-auto-import-webpack]⠼ building... [ember-auto-import-webpack]⠴ building... [ember-auto-import-webpack]⠦ building... [ember-auto-import-webpack]⠧ building... [ember-auto-import-webpack]⠇ building... [ember-auto-import-webpack]⠏ building... [ember-auto-import-webpack]⠋ building... [ember-auto-import-webpack]⠙ building... [ember-auto-import-webpack]⠹ building... [ember-auto-import-webpack]⠸ building... [ember-auto-import-webpack]⠼ building... [ember-auto-import-webpack]⠴ building... [ember-auto-import-webpack]⠦ building... [ember-auto-import-webpack]⠧ building... [ember-auto-import-webpack]⠇ building... [ember-auto-import-webpack]⠏ building... [ember-auto-import-webpack]⠋ building... [ember-auto-import-webpack]⠙ building... [ember-auto-import-webpack]⠹ building... [ember-auto-import-webpack]⠸ building... [ember-auto-import-webpack]⠼ building... [ember-auto-import-webpack]⠴ building... [ember-auto-import-webpack]⠦ building... [ember-auto-import-webpack]⠧ building... [TerserWriter]⠇ building... [TerserWriter]⠏ building... [TerserWriter]⠋ building... [TerserWriter]⠙ building... [TerserWriter]⠹ building... [TerserWriter]⠸ building... [TerserWriter]⠼ building... [TerserWriter]⠴ building... [TerserWriter]⠦ building... [TerserWriter]⠧ building... [TerserWriter]⠇ building... [TerserWriter]⠏ building... [TerserWriter]⠋ building... [TerserWriter]⠙ building... [TerserWriter]⠹ building... [TerserWriter]⠸ building... [TerserWriter]⠼ building... [TerserWriter]⠴ building... [TerserWriter]⠦ building... [TerserWriter]⠧ building... [TerserWriter]⠇ building... [TerserWriter]⠏ building... [TerserWriter]⠋ building... [TerserWriter]⠙ building... [TerserWriter]⠹ building... [TerserWriter]⠸ building... [TerserWriter]⠼ building... [TerserWriter]⠴ building... [TerserWriter]⠦ building... [TerserWriter]⠧ building... [TerserWriter]⠇ building... [TerserWriter]⠏ building... [TerserWriter]⠋ building... [TerserWriter]⠙ building... [TerserWriter]⠹ building... [TerserWriter]⠸ building... [TerserWriter]⠼ building... [TerserWriter]⠴ building... [TerserWriter]⠦ building... [TerserWriter]⠧ building... [TerserWriter]⠇ building... [TerserWriter]⠏ building... [TerserWriter]⠋ building... [TerserWriter]⠙ building... [TerserWriter]⠹ building... [TerserWriter]⠸ building... [TerserWriter]⠼ building... [TerserWriter]⠴ building... [TerserWriter]⠦ building... [TerserWriter]⠧ building... [TerserWriter]⠇ building... [TerserWriter]⠏ building... [TerserWriter]⠋ building... [TerserWriter]⠙ building... [TerserWriter]⠹ building... [TerserWriter]⠸ building... [TerserWriter]⠼ building... [TerserWriter]⠴ building... [TerserWriter]⠦ building... [TerserWriter]⠧ building... [TerserWriter]⠇ building... [TerserWriter]⠏ building... [TerserWriter]⠋ building... [TerserWriter]⠙ building... [TerserWriter]⠹ building... [TerserWriter]⠸ building... [TerserWriter]⠼ building... [TerserWriter]⠴ building... [TerserWriter]⠦ building... [TerserWriter]⠧ building... [TerserWriter]⠇ building... [TerserWriter]⠏ building... [TerserWriter]⠋ building... [TerserWriter]⠙ building... [TerserWriter]⠹ building... [TerserWriter]⠸ building... [TerserWriter]⠼ building... [TerserWriter]⠴ building... [TerserWriter]⠦ building... [TerserWriter]⠧ building... [TerserWriter]⠇ building... [TerserWriter]⠏ building... [TerserWriter]⠋ building... [TerserWriter]⠙ building... [TerserWriter]⠹ building... [TerserWriter]⠸ building... [TerserWriter]⠼ building... [TerserWriter]⠴ building... [TerserWriter]⠦ building... [TerserWriter]⠧ building... [TerserWriter]⠇ building... [TerserWriter]⠏ building... [TerserWriter]⠋ building... [TerserWriter]⠙ building... [TerserWriter]⠹ building... [TerserWriter]⠸ building... [TerserWriter]⠼ building... [TerserWriter]⠴ building... [TerserWriter]⠦ building... [TerserWriter]⠧ building... [TerserWriter]⠇ building... [TerserWriter]⠏ building... [TerserWriter]⠋ building... [TerserWriter]Killed
    Sep 01 00:36:17error Command failed with exit code 137.
    Sep 01 00:36:17info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
    Sep 01 00:36:18==> Creating directories
    Sep 01 00:36:18==> Configuring discourse
    Sep 01 00:36:18==> Creating symlinks for built-in plugins
    Sep 01 00:36:18==> Changing permissions
    Sep 01 00:36:18==> Migrating database
    Sep 01 00:36:19fatal: not a git repository (or any of the parent directories): .git
    Sep 01 00:36:20=> Healtheck error: Error: connect ECONNREFUSED 172.18.16.86:80
    Sep 01 00:36:21==> Pre-compiling assets
    Sep 01 00:36:22fatal: not a git repitory (or any of the parent directories): .git
    Sep 01 00:36:22yarn run v1.22.
    

    I use Cloudron with Gandi & Hetzner

    girishG 1 Reply Last reply
    0
  • girishG Offline
    girishG Offline
    girish Staff
    replied to jdaviescoates on last edited by
    #2

    @jdaviescoates so nothing happens after that build line in the logs? Usually asset building is a heavy CPU operation, so maybe it's just taking time?

    jdaviescoatesJ 1 Reply Last reply
    0
  • jdaviescoatesJ Offline
    jdaviescoatesJ Offline
    jdaviescoates
    replied to girish on last edited by jdaviescoates
    #3

    @girish said in Discourse app stuck in Not Responding state after full Cloudron restore (dry run):

    @jdaviescoates so nothing happens after that build line in the logs?

    No, nothing happens. Well, it just keeps trying to do the same thing over and over again but failing. I originally tried to post more of the logs but the forum wouldn't allow such a long post.

    @girish said in Discourse app stuck in Not Responding state after full Cloudron restore (dry run):

    Usually asset building is a heavy CPU operation, so maybe it's just taking time?

    This is on a dedicated server with plenty of spare CPU. I does use a lot, but it doesn't achieve anything. I've left it all night, still exactly the same.

    The only thing that's changed overnight is that now all my instances of WordPress are in an error state too! (this is all on the dry run restore so no huge deal other than it's slowing down the migrate and at this rate I'll need to do another back-up of the current server and start again...)

    The Discourse did seemingly keep running out of memory though, so perhaps that was it. I've just whacked it up and am trying again...

    Oh, but now I seem to be running out of disk space too 😞

    But that's odd because here is the current server showing I'm using 317.92 GB:

    cfa2d104-720e-434c-b207-e8bf74c8ff08-image.png

    But here is the new server saying I'm somehow now using 429.56 GB!

    f4f4b66f-5248-406b-a2f9-a430c304abf6-image.png

    Somehow the full restore on the new server is taking up over 100GB more space - how is that possible?!?

    Looks like the culprit is the "Everything else" which has somehow massively grown by nearly 130GB! WTF?!?

    Current 27.87 GB:

    72c118e6-ae7f-4800-9271-a2a44a8af6ca-image.png

    New 156.94 GB:

    38f41341-be61-4ca3-bfc0-d2434cb2f7cc-image.png

    Perhaps something to do with a failed partial backup the dry run tried to do overnight? 🤷

    Seeing this error message:

    Error piping http://172.18.30.2:3000/databases/db737de757d2b64357ad482b3b33ce982d/backup?access_token=b13348ff73804eb6301b8a966fca439b94ea1eee9cbe3cfbcc6b67b9cd4baa3299c7465706acf44e9897d5cb6534badf50ecb7eeb311f1a729cf25743f526323e9cda42f982db27018d16ee7dc145734933e4082713cca5374c0668b1ce8cc1ff136af5adbfa0db608707087b616fb82b1cba0445cff453adf5ea2dc42e4f083 to /home/yellowtent/appsdata/737de757-d2b6-4357-ad48-2b3b33ce982d/postgresqldump: ENOSPC: no space left on device, write
    

    Edit: Oh! But while I was noticing all that, the instance of Discourse did finally manage to start after I gave it 6.5GB RAM!

    So now just need to work out why all the WordPress app have crashed and why "Everything else" is so massive...

    Ah, it's because of the disk space, seeing a load of these:

    Docker Error: Not enough disk space to pull docker image
    

    I use Cloudron with Gandi & Hetzner

    1 Reply Last reply
    0
  • jdaviescoatesJ Offline
    jdaviescoatesJ Offline
    jdaviescoates
    wrote on last edited by
    #4

    OK, so now everything is running again, I've just got this disk space issue...

    How can I work out why "Everything" else is taking up nearly 130 GB more space that it was before?

    I use Cloudron with Gandi & Hetzner

    1 Reply Last reply
    0
  • jdaviescoatesJ Offline
    jdaviescoatesJ Offline
    jdaviescoates
    wrote on last edited by jdaviescoates
    #5

    Seems the Disk Usage graph is wrong, trying to refresh it but it's taking an age...

    c94210f8-6cd6-4f7b-86b1-5b55ddc5751c-image.png

    The little refresh button has been turning for yonks now, nothing seems to be happening.

    But if I login via SSH and do df -h I see this:

    root@Ubuntu-2204-jammy-amd64-base ~ # df -h
    Filesystem                           Size  Used Avail Use% Mounted on
    tmpfs                                6.3G  6.5M  6.3G   1% /run
    /dev/md2                             436G  271G  143G  66% /
    

    So seemingly there is actually plenty of space left of the drive (I just did a docker prune -a - although it told me that free up 0B - and Cleaned Up backups, so maybe that did it?)

    Hard to tell if anything has changed without the Disk Usage graph refreshing...

    I use Cloudron with Gandi & Hetzner

    1 Reply Last reply
    0
  • jdaviescoatesJ Offline
    jdaviescoatesJ Offline
    jdaviescoates
    wrote on last edited by
    #6

    How can I force the Disk Usage graphs to refresh? I clicked refresh like half an hour ago or something and nothing has happened, it's just going around and around...

    I use Cloudron with Gandi & Hetzner

    jdaviescoatesJ 1 Reply Last reply
    0
  • jdaviescoatesJ Offline
    jdaviescoatesJ Offline
    jdaviescoates
    replied to jdaviescoates on last edited by
    #7

    @jdaviescoates said in Discourse app stuck in Not Responding state after full Cloudron restore (dry run):

    How can I force the Disk Usage graphs to refresh? I clicked refresh like half an hour ago or something and nothing has happened, it's just going around and around...

    It finally did it and now is showing this:

    d01307b3-bbfa-4d7e-907d-d95a297f20e1-image.png

    I wonder what caused Everything else to balloon so much previously?!? 🤷

    I use Cloudron with Gandi & Hetzner

    girishG 1 Reply Last reply
    0
  • girishG Offline
    girishG Offline
    girish Staff
    replied to jdaviescoates on last edited by
    #8

    @jdaviescoates phew, glad everything is back. Not sure what caused things to balloon up so much though.

    jdaviescoatesJ 1 Reply Last reply
    1
  • jdaviescoatesJ Offline
    jdaviescoatesJ Offline
    jdaviescoates
    replied to girish on last edited by
    #9

    @girish said in Discourse app stuck in Not Responding state after full Cloudron restore (dry run):

    Not sure what caused things to balloon up so much though.

    Yeah, that's a bit of a mystery 🤷

    I use Cloudron with Gandi & Hetzner

    1 Reply Last reply
    0

  • Login

  • Don't have an account? Register

  • Login or register to search.
  • First post
    Last post
0
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks