Cloudron stuck in restart loop since 8.x

[im-fabian]

Hi,

we have some serious issues with our Cloudron Server since 8.2.x. To get a glimpse this is our Event log. And by looking into syslog and boxlog I am unable to get a clue why this happens and what fails. This Issue was already there with 8.2.3 or maybe earlier. We have those since somewhere in january.

Symtoms:
Box is unrseponsive, single sign on often fails. Backups often fail with "Task was stopped because the server was restarted or crashed".

Troubleshoot Output:

root@my ~ # cloudron-support --troubleshoot
Vendor: Hetzner Product:
Linux: 6.8.0-52-generic
Ubuntu: noble 24.04
Processor: AMD Ryzen 9 5950X 16-Core Processor
BIOS AMD Ryzen 9 5950X 16-Core Processor             Unknown CPU @ 3.4GHz x 32
RAM: 131817448KB
Disk: /dev/md2        2.9T
[OK]	node version is correct
[OK]	IPv6 is enabled and public IPv6 address is working
[OK]	docker is running
[OK]	docker version is correct
[OK]	MySQL is running
[OK]	nginx is running
[OK]	dashboard cert is valid
[OK]	dashboard is reachable via loopback
[OK]	box v8.2.4 is running
[OK]	netplan is good
[OK]	DNS is resolving via systemd-resolved
[OK]	Dashboard is reachable via domain name
[OK]	Domain XXX.XX is valid and has not expired
[OK]	unbound is running

mySQL

root@my ~ # systemctl status mysql
● mysql.service - MySQL Community Server
     Loaded: loaded (/usr/lib/systemd/system/mysql.service; enabled; preset: enabled)
     Active: active (running) since Sun 2025-02-23 15:39:29 CET; 5min ago
    Process: 1175 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
   Main PID: 1243 (mysqld)
     Status: "Server is operational"
      Tasks: 43 (limit: 154388)
     Memory: 289.7M (peak: 290.3M)
        CPU: 6.854s
     CGroup: /system.slice/mysql.service
             └─1243 /usr/sbin/mysqld

Feb 23 15:39:29 my systemd[1]: Starting mysql.service - MySQL Community Server...
Feb 23 15:39:29 my systemd[1]: Started mysql.service - MySQL Community Server.

No Errors or Warnings in /var/log/mysql/error.log except for Self signed CA and "''mysql_native_password' is deprecated and will be removed in a future release"

NGINX
nignx -t rund without any errors.

systemd-resolved: ok

• is running
• host www.cloudron.io 127.0.0.53 works
• 127.0.0.53 is set as nameserver under /etc/resolv.conf

NAT

• No NAT used.

/home/yellowtent/platformdata/logs/box.log (at relevant time)

2025-02-23T14:41:00.078Z box:apphealthmonitor app health: 11 running / 2 stopped / 2 unresponsive
2025-02-23T14:41:10.070Z box:apphealthmonitor app health: 11 running / 2 stopped / 2 unresponsive
2025-02-23T14:41:20.082Z box:apphealthmonitor app health: 11 running / 2 stopped / 2 unresponsive
2025-02-23T14:41:30.077Z box:apphealthmonitor app health: 11 running / 2 stopped / 2 unresponsive
2025-02-23T14:41:40.073Z box:apphealthmonitor app health: 11 running / 2 stopped / 2 unresponsive
2025-02-23T14:41:50.623Z box:apphealthmonitor app health: 11 running / 2 stopped / 2 unresponsive
2025-02-23T14:42:00.091Z box:apphealthmonitor app health: 11 running / 2 stopped / 2 unresponsive
2025-02-23T14:42:10.078Z box:apphealthmonitor app health: 11 running / 2 stopped / 2 unresponsive
2025-02-23T14:42:20.089Z box:apphealthmonitor app health: 11 running / 2 stopped / 2 unresponsive
2025-02-23T14:42:30.092Z box:apphealthmonitor app health: 11 running / 2 stopped / 2 unresponsive
2025-02-23T14:42:40.090Z box:apphealthmonitor app health: 11 running / 2 stopped / 2 unresponsive
2025-02-23T14:42:50.067Z box:apphealthmonitor app health: 11 running / 2 stopped / 2 unresponsive
2025-02-23T14:43:00.091Z box:apphealthmonitor app health: 11 running / 2 stopped / 2 unresponsive
2025-02-23T14:43:10.087Z box:apphealthmonitor app health: 11 running / 2 stopped / 2 unresponsive
2025-02-23T14:43:20.058Z box:apphealthmonitor app health: 11 running / 2 stopped / 2 unresponsive
2025-02-23T14:43:32.152Z box:server ==========================================
2025-02-23T14:43:32.152Z box:server            Cloudron 8.2.4
2025-02-23T14:43:32.152Z box:server ==========================================
2025-02-23T14:43:32.153Z box:platform initialize: start platform
2025-02-23T14:43:32.174Z box:tasks stopAllTasks: stopping all tasks
2025-02-23T14:43:32.174Z box:shell tasks /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
2025-02-23T14:43:32.195Z box:shell All tasks stopped
2025-02-23T14:43:32.203Z box:locks releaseAll: all locks released
2025-02-23T14:43:32.205Z box:reverseproxy writeDashboardConfig: writing dashboard config for DOMAIN.TLD
2025-02-23T14:43:32.208Z box:shell reverseproxy: openssl x509 -in /home/yellowtent/platformdata/nginx/cert/my.DOMAIN.TLD.cert -noout -ocsp_uri
2025-02-23T14:43:32.228Z box:shell reverseproxy /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
2025-02-23T14:43:32.289Z box:shell 2025/02/23 15:43:32 [notice] 11425#11425: signal process started
2025-02-23T14:43:32.453Z box:platform onActivated: starting post activation services
2025-02-23T14:43:32.453Z box:platform startInfra: checking infrastructure
2025-02-23T14:43:32.453Z box:platform startInfra: infra is uptodate at version 49.8.0
2025-02-23T14:43:32.453Z box:platform onInfraReady: platform is ready. infra changed: false
2025-02-23T14:43:32.453Z box:apps schedulePendingTasks: scheduling app tasks
2025-02-23T14:43:32.471Z box:apptaskmanager started
2025-02-23T14:43:32.471Z box:cron startJobs: starting cron jobs with hour 21 and minute 12
2025-02-23T14:43:32.483Z box:cron backupPolicyChanged: schedule 00 00 23 * * * (Europe/Berlin)
2025-02-23T14:43:32.485Z box:cron autoupdatePatternChanged: pattern - 00 00 7,8 * * 4 (Europe/Berlin)
2025-02-23T14:43:32.486Z box:cron Dynamic DNS setting changed to false
2025-02-23T14:43:32.487Z box:dockerproxy start: listening on 172.18.0.1:3003
2025-02-23T14:43:35.908Z box:user notifyLoginLocation: uid-a672450b-0cbe-4b28-aaf3-f33469456742 2.202.126.28 Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:135.0) Gecko/20100101 Firefox/135.0
2025-02-23T14:43:37.649Z box:server ==========================================
2025-02-23T14:43:37.650Z box:server            Cloudron 8.2.4
2025-02-23T14:43:37.650Z box:server ==========================================
2025-02-23T14:43:37.650Z box:platform initialize: start platform
2025-02-23T14:43:37.672Z box:tasks stopAllTasks: stopping all tasks
2025-02-23T14:43:37.672Z box:shell tasks /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
2025-02-23T14:43:37.693Z box:shell All tasks stopped
2025-02-23T14:43:37.702Z box:locks releaseAll: all locks released
2025-02-23T14:43:37.706Z box:reverseproxy writeDashboardConfig: writing dashboard config for DOMAIN.TLD
2025-02-23T14:43:37.708Z box:shell reverseproxy: openssl x509 -in /home/yellowtent/platformdata/nginx/cert/my.DOMAIN.TLD.cert -noout -ocsp_uri
2025-02-23T14:43:37.728Z box:shell reverseproxy /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
2025-02-23T14:43:37.790Z box:shell 2025/02/23 15:43:37 [notice] 11487#11487: signal process started
2025-02-23T14:43:37.955Z box:platform onActivated: starting post activation services
2025-02-23T14:43:37.955Z box:platform startInfra: checking infrastructure
2025-02-23T14:43:37.955Z box:platform startInfra: infra is uptodate at version 49.8.0
2025-02-23T14:43:37.956Z box:platform onInfraReady: platform is ready. infra changed: false
2025-02-23T14:43:37.956Z box:apps schedulePendingTasks: scheduling app tasks
2025-02-23T14:43:37.975Z box:apptaskmanager started
2025-02-23T14:43:37.975Z box:cron startJobs: starting cron jobs with hour 21 and minute 12
2025-02-23T14:43:37.990Z box:cron backupPolicyChanged: schedule 00 00 23 * * * (Europe/Berlin)
2025-02-23T14:43:37.992Z box:cron autoupdatePatternChanged: pattern - 00 00 7,8 * * 4 (Europe/Berlin)
2025-02-23T14:43:37.993Z box:cron Dynamic DNS setting changed to false
2025-02-23T14:43:37.994Z box:dockerproxy start: listening on 172.18.0.1:3003
2025-02-23T14:43:40.087Z box:apphealthmonitor app health: 11 running / 2 stopped / 2 unresponsive
2025-02-23T14:43:40.990Z box:user notifyLoginLocation: uid-a672450b-0cbe-4b28-aaf3-f33469456742 2.202.126.28 Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:135.0) Gecko/20100101 Firefox/135.0
2025-02-23T14:43:42.652Z box:server ==========================================
2025-02-23T14:43:42.653Z box:server            Cloudron 8.2.4
2025-02-23T14:43:42.653Z box:server ==========================================
2025-02-23T14:43:42.653Z box:platform initialize: start platform
2025-02-23T14:43:42.675Z box:tasks stopAllTasks: stopping all tasks
2025-02-23T14:43:42.675Z box:shell tasks /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
2025-02-23T14:43:42.696Z box:shell All tasks stopped
2025-02-23T14:43:42.706Z box:locks releaseAll: all locks released
2025-02-23T14:43:42.707Z box:reverseproxy writeDashboardConfig: writing dashboard config for DOMAIN.TLD
2025-02-23T14:43:42.709Z box:shell reverseproxy: openssl x509 -in /home/yellowtent/platformdata/nginx/cert/my.DOMAIN.TLD.cert -noout -ocsp_uri
2025-02-23T14:43:42.729Z box:shell reverseproxy /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
2025-02-23T14:43:42.789Z box:shell 2025/02/23 15:43:42 [notice] 11544#11544: signal process started
2025-02-23T14:43:42.954Z box:platform onActivated: starting post activation services
2025-02-23T14:43:42.954Z box:platform startInfra: checking infrastructure
2025-02-23T14:43:42.954Z box:platform startInfra: infra is uptodate at version 49.8.0
2025-02-23T14:43:42.954Z box:platform onInfraReady: platform is ready. infra changed: false
2025-02-23T14:43:42.954Z box:apps schedulePendingTasks: scheduling app tasks
2025-02-23T14:43:42.973Z box:apptaskmanager started
2025-02-23T14:43:42.973Z box:cron startJobs: starting cron jobs with hour 21 and minute 12
2025-02-23T14:43:42.988Z box:cron backupPolicyChanged: schedule 00 00 23 * * * (Europe/Berlin)
2025-02-23T14:43:42.990Z box:cron autoupdatePatternChanged: pattern - 00 00 7,8 * * 4 (Europe/Berlin)
2025-02-23T14:43:42.991Z box:cron Dynamic DNS setting changed to false
2025-02-23T14:43:42.992Z box:dockerproxy start: listening on 172.18.0.1:3003
2025-02-23T14:43:50.107Z box:apphealthmonitor app health: 11 running / 2 stopped / 2 unresponsive

[timconsidine]

What is the VPS ram ?

[im-fabian]

126 GB RAM (~24 GB used) + 16 GB SWAP (0,2 GB used)

[timconsidine]

Well that’s a beefy box (wasn’t sure if ram was low but not so)

[robi]

Sounds like the power is going out

[joseph]

@im-fabian it looks like the code is crashing somewhere and the crash error is not logged.

You are able to access the apps itself, correct? Since nginx/mysql are running, apps should be fine. It is the dashboard which is crashing non-stop .

Can you try this:

• Edit /home/yellowtent/box/src/users.js
• Just add a "return;" here - https://git.cloudron.io/platform/box/-/blob/master/src/users.js?ref_type=heads#L752 . Mostly because all the time code crashes in notifyLoginLocation
• systemctl restart box

[im-fabian]

Hi,
yeah, die apps themself work – as long as they do not need Cloudron for SSO.

We added return but this does not change the behaviour, we still have multiple crashes a hour.

[joseph]

@im-fabian can you run /home/yellowtent/box/setup/start.sh and see if that helps? Doing that runs the db migrations

[im-fabian]

@joseph We already did that with no change.

[joseph]

@im-fabian can you reach out at support@cloudron.io ? I think we have debug further with access to the server. thanks!