Seems like my server may be affected by the 9.1.7 update, lots of weird problems

[scooke]

At first I noticed my email not connecting. I tried one of my sites and it didn't load. I went to the dashboard and everything was suuppperr slow. I rebooted from the dashboard, and then ran sudo cloudron-support --troubleshoot and showed some services Failed. Mail wasn't running so I ran docker start mail, and that got it going, but still no apps, they were all Pending. I've rebooted at least 4 times now and most of the same things aren't working. I've just checked the Services in the daashboard and all the DBs, Mongo and FileManager are white or grey, and Redis are red. I've run sudo cloudron-support --troubleshoot and this is what it shows:

Vendor: QEMU Product: Standard PC (Q35 + ICH9, 2009)
Linux: 5.15.0-177-generic
Ubuntu: jammy 22.04
Cloudron: 9.1.7
Execution environment: kvm
Processor: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz x 12
RAM: 49327400KB
Disk: /dev/sda2       392G
[OK]	node version is correct
[OK]	IPv6 is enabled and public IPv6 address is working
[OK]	docker is running
[OK]	docker version is correct
[OK]	MySQL is running
[OK]	netplan is good
[OK]	DNS is resolving via systemd-resolved
[OK]	unbound is running
[OK]	nginx is running
[OK]	dashboard cert is valid
[OK]	dashboard is reachable via loopback
[OK]	No pending database migrations
[WARN]	Service 'mysql' is not running (may be lazy-stopped)
[WARN]	Service 'postgresql' is not running (may be lazy-stopped)
[WARN]	Service 'mongodb' is not running (may be lazy-stopped)
[OK]	Service 'mail' is running and healthy
[FAIL]	Service 'graphite' container is not running (state: exited)!
[FAIL]	Service 'sftp' container is not running (state: exited)!
[OK]	box v9.1.7 is running
[OK]	Dashboard is reachable via domain name
[OK]	Domain toutdo.com is valid and has not expired

Pressing Restart on those Services in teh Dashboard has gotten most of them running except Redis, MongoDB and MySQL. The logs for Cloudron Service say all the apps are waiting for 448.684 to update health. Not one app is running this time either whereas after earlier restarts some of them did.

Help!

[scooke]

I've seen a few more apps start up on their own, seemingly. And I've been able to manually start some other apps using either the Restart button on the app in the dashboard, or on the cli using sudo docker run -it cloudron/com.libretranslate.cloudronapp:xxxxxxxxxxxxxx, but that last cli approach only works if I can see which images are running using sudo docker ps so if an app is stalled and not running I can't start it from the cli.

sudo cloudron-support --troubleshoot tells me everything is fine except
[WARN] Service 'mongodb' is not running (may be lazy-stopped).

2026-05-11T02:25:57.731Z apphealthmonitor: app health: 34 running / 11 stopped / 28 unresponsive is the result from sudo tail -f /home/yellowtent/platformdata/logs/box.log. What is going on??

[nebulon]

So what is the current situation? Are all services up besides mongodb (which may be fine, since very few apps depend on that)?

Since you mentioned that everything was very slow, possibly things were busy (re)starting up.

Also since at least in the initial post, graphite and sftp services were not up, apps will also not start up by the Cloudron platform as it has reached a healthy state yet. Is this still the case?

If not, what errors are shown in the app itself if you restart one of the app which are down?

[scooke]

Some apps are running. When I click on some labelled "Not Responding" its dashboard won't load, I can see in the logs May 11 09:08:09 shell: df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/66c962bb-24fa-4237-9166-459875c74e1b/data for that app. Then when I click back, the main dashboard won't load! This longer bit seemed to happen too when I tried to click to the unresponsive app's dashboard:

ay 11 09:07:50 apphealthmonitor: app health: 36 running / 11 stopped / 26 unresponsive
May 11 09:07:50 shell: df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/03233c73-df43-43ec-82ba-0518542ea09a/data
May 11 09:07:51 apphealthmonitor: app health: 36 running / 11 stopped / 26 unresponsive
May 11 09:07:51 shell: df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/040b01cf-5c14-4db5-ba54-83f3356cb466/data
May 11 09:07:52 shell: df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/05460bf6-5d11-4b75-b467-8725d7c07113/data
May 11 09:07:54 shell: df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/05b7c6ea-7452-4505-8345-46903fa89f1f/data
May 11 09:07:58 shell: df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/09c4a0ac-8f68-4d66-9466-634fed6b8e79/data
May 11 09:07:58 {
May 11 09:07:58   path: '/api/v1/system/filesystems',
May 11 09:07:58   status: 500,
May 11 09:07:58   error: ServerError [ServiceUnavailableError]: Response timeout
May 11 09:07:58       at IncomingMessage.<anonymous> (/home/yellowtent/box/node_modules/connect-timeout/index.js:84:8)
May 11 09:07:58       at IncomingMessage.emit (node:events:508:28)
May 11 09:07:58       at Timeout._onTimeout (/home/yellowtent/box/node_modules/connect-timeout/index.js:49:11)
May 11 09:07:58       at listOnTimeout (node:internal/timers:605:17)
May 11 09:07:58       at process.processTimers (node:internal/timers:541:7) {
May 11 09:07:58     code: 'ETIMEDOUT',
May 11 09:07:58     timeout: 60000
May 11 09:07:58   }
May 11 09:07:58 }
May 11 09:07:59 shell: df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/0daa5ebf-8507-4cdd-a657-01f41c0068fc/data
May 11 09:07:59 shell: df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/113cef81-b36d-45ec-9158-82cf1309db37/data
May 11 09:08:00 appheal

Is there something wrong with my disk?
Last night while troubleshooting the dashboard wouldn't load and I'd get these:

[scooke]

And the browser window would all be black. Macos, Chrome Version 147.0.7727.138 (Official Build) (x86_64)

[nebulon]

receiving response timeout from the backend usually means the box process is too slow to respond, which in turn may hint at a system overload. But since one is also failing when querying the fileystem routes, maybe you have a locked up filessytem mounted which makes the system hang?

Does a normal df -h via SSH work fine and are all mountpoints healthy?

[scooke]

Seems like it. I have lots of the below:

overlay                                    709G  281G  392G  42% /var/lib/docker/overlay2/42dc09e1ab2f10ed432cae69adfe96136fd88c3a605b4bc39f823c05cc53b679/merged
overlay                                    709G  281G  392G  42% /var/lib/docker/overlay2/3ac8dfcbe628dcdf82236861f36b799f4392830bad9203669741cb2d9f1a0767/merged
overlay                                    709G  281G  392G  42% /var/lib/docker/overlay2/74626746866ed86941b6e48831acf5f38a2feaa8297c3c4ad6851d4ec591c169/merged
overlay                                    709G  281G  392G  42% /var/lib/docker/overlay2/1c7f0e1688f8c536f6d6081af1f98b8ba7033658f32a173e0eed1f41d62ae847/merged
tmpfs                                      4.8G  4.0K  4.8G   1% /run/user/1000
overlay                                    709G  281G  392G  42% /var/lib/docker/overlay2/4a208722f4e76888c9ff9b9a9a69c1180f28458b075b8bdbc2a81f17029acbeb/merged
overlay                                    709G  281G  392G  42% /var/lib/docker/overlay2/091f8c6db63e863e5046a8ab27351f49f6eef980cadda12dd5d4e06acbbfa491/merged
overlay                                    709G  281G  392G  42% /var/lib/docker/overlay2/358bb3005eb75b4de6a19ac978ca6e9acffaf301e587c3c65278c383ca6588a8/merged

I do have two remote mountpoints which seem fine, but one is at 95%.