"Cloudron is offline. Reconnecting..." after update from 7.4.2 to 7.5.0 on Ubuntu 22.04 machine

[omen]

My system is on Ubuntu 22.04.

Yesterday I updated from 7.4.2 to 7.5.0, but had to go to bed while the backups were taking place. This morning I see "Cloudron is offline. Reconnecting..."

Tried rebooting, but no effect. Followed steps on troubleshooting page, but seem to be getting nowhere.

box, nginx, etc all seem to be up and good, but docker is down and refuses to start.

Here are the results of various commands and log files:

Checking /home/yellowtent/platformdata/logs/box.log shows this message repeated again and again:

2023-07-01T06:07:00.043Z box:cron BoxError: connect ECONNREFUSED /var/run/docker.sock
    at Object.getEvents (/home/yellowtent/box/src/docker.js:611:22)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async processDockerEvents (/home/yellowtent/box/src/apphealthmonitor.js:133:20)
    at async Object.run (/home/yellowtent/box/src/apphealthmonitor.js:189:5)

Followed by multiple iterations of this at the end:

2023-07-01T17:55:33.990Z box:server ==========================================
2023-07-01T17:55:33.991Z box:server            Cloudron 7.4.2  
2023-07-01T17:55:33.991Z box:server ==========================================
2023-07-01T17:55:34.019Z box:settings initCache: pre-load settings
2023-07-01T17:55:34.030Z box:tasks stopAllTasks: stopping all tasks
2023-07-01T17:55:34.030Z box:shell stopTask spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
2023-07-01T17:55:34.061Z box:shell stopTask (stdout): All tasks stopped

2023-07-01T17:55:34.062Z box:reverseproxy writeDashboardConfig: writing admin config for wtrvz.com
Cloudron is up and running. Logs are at /home/yellowtent/platformdata/logs/box.log
2023-07-01T17:55:34.090Z box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
2023-07-01T17:55:34.268Z box:cloudron onActivated: running post activation tasks
2023-07-01T17:55:34.269Z box:platform initializing platform
2023-07-01T17:55:34.269Z box:platform platform is uptodate at version 49.4.0
2023-07-01T17:55:34.269Z box:platform onPlatformReady: platform is ready. infra changed: false
2023-07-01T17:55:34.270Z box:apps schedulePendingTasks: scheduling app tasks
2023-07-01T17:55:34.279Z box:cron startJobs: starting cron jobs with hour 23 and minute 2
2023-07-01T17:55:34.305Z box:cron backupConfigChanged: schedule 00 00 23 * * * (America/New_York)
2023-07-01T17:55:34.308Z box:cron autoupdatePatternChanged: pattern - 00 00 1,3,5,23 * * * (America/New_York)
2023-07-01T17:55:34.309Z box:cron Dynamic DNS setting changed to false
2023-07-01T17:55:34.310Z box:dockerproxy startDockerProxy: started proxy on port 3003
Error: listen EADDRNOTAVAIL: address not available 172.18.0.1:3003
    at Server.setupListenHandle [as _listen2] (node:net:1446:21)
    at listenInCluster (node:net:1511:12)
    at doListen (node:net:1660:7)
    at processTicksAndRejections (node:internal/process/task_queues:84:21)

Trying to restart docker daemon just fails.

root@Cloudron2:~# systemctl status docker
× docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/docker.service.d
             └─cloudron.conf, custom.conf
     Active: failed (Result: exit-code) since Sat 2023-07-01 17:38:18 UTC; 8min ago
TriggeredBy: × docker.socket
       Docs: https://docs.docker.com
    Process: 1100 ExecStart=/usr/bin/dockerd -H fd:// --log-driver=journald --exec-opt native.cgroupdriver=cgroupfs --storage-driver=overlay2>
   Main PID: 1100 (code=exited, status=1/FAILURE)
        CPU: 101ms

Jul 01 17:38:16 Cloudron2 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Jul 01 17:38:16 Cloudron2 systemd[1]: docker.service: Failed with result 'exit-code'.
Jul 01 17:38:16 Cloudron2 systemd[1]: Failed to start Docker Application Container Engine.
Jul 01 17:38:18 Cloudron2 systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Jul 01 17:38:18 Cloudron2 systemd[1]: Stopped Docker Application Container Engine.
Jul 01 17:38:18 Cloudron2 systemd[1]: docker.service: Start request repeated too quickly.
Jul 01 17:38:18 Cloudron2 systemd[1]: docker.service: Failed with result 'exit-code'.
Jul 01 17:38:18 Cloudron2 systemd[1]: Failed to start Docker Application Container Engine.

root@Cloudron2:~# docker info
Client:
 Context:    default
 Debug Mode: false

Server:
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info

root@Cloudron2:~# journalctl -xeu docker.service
░░ An ExecStart= process belonging to unit docker.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Jul 01 17:47:24 Cloudron2 systemd[1]: docker.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit docker.service has entered the 'failed' state with result 'exit-code'.
Jul 01 17:47:24 Cloudron2 systemd[1]: Failed to start Docker Application Container Engine.
░░ Subject: A start job for unit docker.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit docker.service has finished with a failure.
░░ 
░░ The job identifier is 39437 and the job result is failed.
Jul 01 17:47:26 Cloudron2 systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ Automatic restarting of the unit docker.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
Jul 01 17:47:26 Cloudron2 systemd[1]: Stopped Docker Application Container Engine.
░░ Subject: A stop job for unit docker.service has finished
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A stop job for unit docker.service has finished.
░░ 
░░ The job identifier is 39729 and the job result is done.
Jul 01 17:47:26 Cloudron2 systemd[1]: docker.service: Start request repeated too quickly.
Jul 01 17:47:26 Cloudron2 systemd[1]: docker.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit docker.service has entered the 'failed' state with result 'exit-code'.
Jul 01 17:47:26 Cloudron2 systemd[1]: Failed to start Docker Application Container Engine.
░░ Subject: A start job for unit docker.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit docker.service has finished with a failure.
░░ 
░░ The job identifier is 39729 and the job result is failed.

Any ideas?

[robi]

disk space?

[omen]

@robi I don't think so. When I checked just yesterday it was all good. I have all the docker images on a separate large volume with only about 50% usage.

[omen]

root@Cloudron2:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           794M  1.1M  793M   1% /run
/dev/vda1        49G   14G   35G  29% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/vda15      105M  5.3M  100M   5% /boot/efi
/dev/sda        100G   45G   50G  48% /mnt/cloudron_volume_a
tmpfs           794M  4.0K  794M   1% /run/user/0

[robi]

try reboot.

[omen]

I tried that earlier, and again just now... but no change

[robi]

then best to email support@ and make sure to have remote support enabled.

[omen]

Good idea. Email sent. Thanks

[girish]

It seems docker is down but will look into your email.

[girish]

@omen I guess you were using some custom docker options (as you mentioned earlier in the post).

Jul 02 03:19:26 Cloudron2 systemd[1]: Starting Docker Application Container Engine...
Jul 02 03:19:26 Cloudron2 dockerd[717209]: Flag --graph has been deprecated, Use --data-root instead
Jul 02 03:19:26 Cloudron2 dockerd[717209]: the "graph" config file option is deprecated; use "data-root" instead
Jul 02 03:19:26 Cloudron2 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Jul 02 03:19:26 Cloudron2 systemd[1]: docker.service: Failed with result 'exit-code'.
Jul 02 03:19:26 Cloudron2 systemd[1]: Failed to start Docker Application Container Engine.

I updated /etc/systemd/system/docker.service.d/custom.conf to use --data-root as it says above. And now it works.

[omen]

Appreciate the help!

Weird. The only manual docker change I remember making was following the instructions for moving content from the main machine to my volume. https://docs.cloudron.io/storage/

[girish]

@omen right.. the docs are outdated. Fixed now.

[ctrl]

Tyvm, saved my day!