Cloudron updated to version 8.03 and stuck on Configuring Queued All Apps

[skeats]

Hello,

My Cloudron server updated to version 8.0.3 overnight and needed to be rebooted. I did the reboot and now the instance is stuck on the redis service for Chatwoot as starting, but it is not starting, which is holding up all the other apps from starting. I need a resolution because this is crazy. My base OS is still Ubuntu Server 22.04.03. Do I need to fix that first? I have some mission critical apps I use on there for the business.

Thanks,

Scott

[skeats]

Here is a log showing the error with Chatwoot not working with redis:

Aug 12 09:36:00 box:services Waiting for redis-87eca661-9dc4-4b00-8366-7cc47feb319e
Aug 12 09:36:00 box:services Attempt 1 failed. Will retry: Network error waiting for redis-87eca661-9dc4-4b00-8366-7cc47feb319e: connect ECONNREFUSED 172.18.0.11:3000
Aug 12 09:36:00 box:services startRedis: stopping and deleting previous redis container redis-87eca661-9dc4-4b00-8366-7cc47feb319e
Aug 12 09:36:10 box:services startRedis: starting redis container redis-87eca661-9dc4-4b00-8366-7cc47feb319e
Aug 12 09:36:10 box:shell startRedis exec: docker run --restart=always -d --name=redis-87eca661-9dc4-4b00-8366-7cc47feb319e --hostname redis-87eca661-9dc4-4b00-8366-7cc47feb319e --label=location=chatwoot.experiencedmg.net --net cloudron --net-alias redis-87eca661-9dc4-4b00-8366-7cc47feb319e --log-driver syslog --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock --log-opt syslog-format=rfc5424 --log-opt tag=redis-87eca661-9dc4-4b00-8366-7cc47feb319e -m 268435456 --memory-swap -1 -e CLOUDRON_REDIS_PASSWORD=7a843b3e461fcaed5c3a7c859eba8dc43f6fbd71ca16594e -e CLOUDRON_REDIS_TOKEN=85725a3ba6628732adcf7e0a7b17e4a220a14ee8df212249 -v /home/yellowtent/platformdata/redis/87eca661-9dc4-4b00-8366-7cc47feb319e:/var/lib/redis --label isCloudronManaged=true --read-only -v /tmp -v /run registry.docker.com/cloudron/redis:3.5.3@sha256:1e1200900c6fb196950531ecec43f400b4fe5e559fac1c75f21e6f0c11885b5f
Aug 12 09:36:11 box:services Waiting for redis-87eca661-9dc4-4b00-8366-7cc47feb319e
Aug 12 09:36:11 box:services Attempt 1 failed. Will retry: Network error waiting for redis-87eca661-9dc4-4b00-8366-7cc47feb319e: connect ECONNREFUSED 172.18.0.11:3000
Aug 12 09:36:15 box:services Attempt 2 failed. Will retry: Error waiting for redis-87eca661-9dc4-4b00-8366-7cc47feb319e. Status code: 401 message: undefined
Aug 12 09:36:16 box:shell restartcollectd /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh collectd
Aug 12 09:36:26 box:services Attempt 2 failed. Will retry: Error waiting for redis-87eca661-9dc4-4b00-8366-7cc47feb319e. Status code: 200 message: process not running
Aug 12 09:36:30 box:services Attempt 3 failed. Will retry: Error waiting for redis-87eca661-9dc4-4b00-8366-7cc47feb319e. Status code: 401 message: undefined
Aug 12 09:36:41 box:services Attempt 3 failed. Will retry: Error waiting for redis-87eca661-9dc4-4b00-8366-7cc47feb319e. Status code: 200 message: process not running
Aug 12 09:36:45 box:services Attempt 4 failed. Will retry: Error waiting for redis-87eca661-9dc4-4b00-8366-7cc47feb319e. Status code: 401 message: undefined
Aug 12 09:36:56 box:services Attempt 4 failed. Will retry: Error waiting for redis-87eca661-9dc4-4b00-8366-7cc47feb319e. Status code: 200 message: process not running
[no timestamp] [ServiceUnavailableError]: Response timeout
[no timestamp] at IncomingMessage.<anonymous> (/home/yellowtent/box/node_modules/connect-timeout/index.js:84:8)
[no timestamp] at IncomingMessage.emit (node:events:518:28)
[no timestamp] at Timeout._onTimeout (/home/yellowtent/box/node_modules/connect-timeout/index.js:49:11)
[no timestamp] at listOnTimeout (node:internal/timers:573:17)
[no timestamp] at process.processTimers (node:internal/timers:514:7) {
[no timestamp] code: 'ETIMEDOUT',
[no timestamp] timeout: 60000
[no timestamp] }
Aug 12 09:37:00 box:services Attempt 5 failed. Will retry: Error waiting for redis-87eca661-9dc4-4b00-8366-7cc47feb319e. Status code: 401 message: undefined
Aug 12 09:37:11 box:services Attempt 5 failed. Will retry: Error waiting for redis-87eca661-9dc4-4b00-8366-7cc47feb319e. Status code: 200 message: process not running
Aug 12 09:37:14 box:shell getSwaps execArgs: swapon ["--noheadings","--raw","--bytes","--show=type,size,used,name"]
Aug 12 09:37:15 box:services Attempt 6 failed. Will retry: Error waiting for redis-87eca661-9dc4-4b00-8366-7cc47feb319e. Status code: 401 message: undefined
Aug 12 09:37:26 box:services Attempt 6 failed. Will retry: Error waiting for redis-87eca661-9dc4-4b00-8366-7cc47feb319e. Status code: 200 message: process not running
Aug 12 09:37:30 box:services Attempt 7 failed. Will retry: Error waiting for redis-87eca661-9dc4-4b00-8366-7cc47feb319e. Status code: 401 message: undefined
Aug 12 09:37:42 box:services Attempt 7 failed. Will retry: Error waiting for redis-87eca661-9dc4-4b00-8366-7cc47feb319e. Status code: 200 message: process not running
Aug 12 09:37:45 box:services Attempt 8 failed. Will retry: Error waiting for redis-87eca661-9dc4-4b00-8366-7cc47feb319e. Status code: 401 message: undefined
Aug 12 09:37:57 box:services Attempt 8 failed. Will retry: Error waiting for redis-87eca661-9dc4-4b00-8366-7cc47feb319e. Status code: 200 message: process not running
Aug 12 09:38:00 box:services Attempt 9 failed. Will retry: Error waiting for redis-87eca661-9dc4-4b00-8366-7cc47feb319e. Status code: 401 message: undefined
Aug 12 09:38:09 box:shell getSwaps execArgs: swapon ["--noheadings","--raw","--bytes","--show=type,size,used,name"]
Aug 12 09:38:12 box:services Attempt 9 failed. Will retry: Error waiting for redis-87eca661-9dc4-4b00-8366-7cc47feb319e. Status code: 200 message: process not running
Aug 12 09:38:13 box:shell getSwaps execArgs: swapon ["--noheadings","--raw","--bytes","--show=type,size,used,name"]
Aug 12 09:38:14 box:shell hasAVX execArgs: grep ["-q","avx","/proc/cpuinfo"]
Aug 12 09:38:14 box:shell statusUnbound execArgs: systemctl ["is-active","unbound"]
Aug 12 09:38:14 box:shell statusNginx execArgs: systemctl ["is-active","nginx"]
Aug 12 09:38:15 box:services Attempt 10 failed. Will retry: Error waiting for redis-87eca661-9dc4-4b00-8366-7cc47feb319e. Status code: 401 message: undefined
Aug 12 09:38:27 box:services Attempt 10 failed. Will retry: Error waiting for redis-87eca661-9dc4-4b00-8366-7cc47feb319e. Status code: 200 message: process not running
Aug 12 09:38:27 box:locker Lock unreleased infra_start
Aug 12 09:38:30 box:services Attempt 11 failed. Will retry: Error waiting for redis-87eca661-9dc4-4b00-8366-7cc47feb319e. Status code: 401 message: undefined
Aug 12 09:38:42 box:services Attempt 11 failed. Will retry: Error waiting for redis-87eca661-9dc4-4b00-8366-7cc47feb319e. Status code: 200 message: process not running
Aug 12 09:38:45 box:services Attempt 12 failed. Will retry: Error waiting for redis-87eca661-9dc4-4b00-8366-7cc47feb319e. Status code: 401 message: undefined

[skeats]

Can I please have a support member such as @girish or @nebulon to review this please. I have tried to fix it, but it is not going well

[joseph]

@skeats can you write to support@cloudron.io and we can have a look? It looks like redis is not starting

[TheNils]

i pretty much have the same problem and logs, an dredis is reporting up and healthy ...
tried to debug it myself but can't get apps to come back online.
i'm reaching support@cloudron.io but it's Sunday.

[skeats]

I was able to fix it by throwing the redis service for Chatwoot into debug mode and then waited 10 minutes, refreshed the dashboard and all the other apps started with no issues. Later the wiki.js app went none responsive so I restarted it and it was fine
Ended up updating my Cloudron server to Ubuntu 24.04 using your instructions and everything is stable again with no issues. I was able to take my backup of Chatwoot and reinstall it and everything is good. Guess version 8.0.3 does not like Ubuntu 22.04.3 very much

[TheNils]

Thanks to the Cloudron team who solved the problem very quickly on a Sunday. I had been struggling with the server for more than 4 hours, and they fixed the issue in 30 minutes!

[skeats]

@TheNils what was done for you to fix your issue? Probably not like the work around that I had to do

[nebulon]

The issue was that docker got stuck in removing the Cloudron network during the starting after the update. Not clear why and when this happens but stopping docker then running docker network rm -f cloudron manually (which will start docker due to socket activation) makes it work...