Server crashes caused by stopped app's runner container stuck in restart loop

[girish]

@mendoksai the container getting created is not a problem. The container is created but not run for stopped apps (i.e docker container create vs docker container run). The issue is also not related to domains (and it's expiry).

I haven't been able to reproduce this issue though.

I think the issue is actually that box code is unable to control docker. Or maybe docker is not running commands properly. For example, Container <id> is restarting, wait until the container is running . This already indicates the stopped app is in incorrect state. The Container has to be in stopped state. Are there any errors in journalctl -u docker -fa ? The rest of the errors like redis not stopping, cron container error is all the same issue of docker not running containers properly.

Error: listen EADDRNOTAVAIL: address not available 172.18.0.1:3003 is similar. Docker is supposed to create the cloudron network in that IP. Can't see how it can be unavailable.

Can you give more information on your environment? Are other apps running properly?

[mendoksai]

Thanks @girish for looking into this.

You're right — this isn't just about the stopped app. After collecting detailed logs, I found multiple containers in incorrect states on every boot:

<appid-1>-runner         Created          (stopped app - Lychee)
<appid-2>                Restarting (1)   (Mattermost)
<appid-3>                Restarting (1)   (Kimai)

The Mattermost container is the main culprit. On boot, it tries to connect to MySQL before it's ready, fails, and enters an infinite restart loop:

error: Failed to ping DB  error="dial tcp 172.18.30.1:3306: connect: connection refused"
Error: failed to initialize platform: cannot create store: error setting up connections

This restart loop seems to degrade Docker networking over time. The Docker journal shows a clear cascade:

1. Boot → Mattermost enters restart loop (MySQL not ready yet)
2. Docker resolver starts failing — first external DNS timeouts, then internal (172.18.0.1:53)
3. Error: listen EADDRNOTAVAIL: address not available 172.18.0.1:3003 on every boot
4. Eventually host MySQL becomes unreachable → full server lockup

For journalctl -u docker, there are no explicit error-level entries from Docker daemon itself — only info level "ignoring event" / "cleaning up dead shim" messages repeating every 5 minutes for the same container, plus error level DNS timeout entries from the resolver.

I've stopped both Mattermost and the Lychee runner for now. Will monitor.

Environment details:

• Cloudron 9.1.3
• Ubuntu 24.04.4 LTS, Kernel 6.8.0-106-generic
• Dedicated Server: 8 CPUs, 32GB RAM
• ~35 containers on the cloudron network
• Docker: Cgroup Driver: cgroupfs, Cgroup Version: 2
• Hardware check by Hetzner: all clean (CPU, disks, NIC)
• Issue started ~3 weeks ago, persisted through kernel 5.15 → 6.8 upgrade

Happy to provide SSH access or full logs if needed.

[mendoksai]

Quick update — I just noticed cloudron-support --troubleshoot was reporting:

[FAIL] Database migrations are pending. Last migration in DB: /20260217120000-mailPasswords-create-table.js

This migration has been pending since Feb 17 — which is exactly when the instability started. I missed this earlier. Just applied it:

cloudron-support --apply-db-migrations
[OK] Database migrations applied successfully

I've also stopped the Mattermost container that was in a restart loop (it was failing to connect to MySQL on boot and never recovering).

Will monitor for the next few days and report back. Fingers crossed this was the missing piece.

[joseph]

@mendoksai said:

Quick update — I just noticed cloudron-support --troubleshoot was reporting:

[FAIL] Database migrations are pending. Last migration in DB: /20260217120000-mailPasswords-create-table.js

This is a bug in the tool and not a real problem. It's fixed in 9.1.5.

[mendoksai]

Happened again. Every a few days.

[nebulon]

Do you by any chance have made some custom modifications to your ubuntu system like applying apt updates or so?

[mendoksai]

Yes, I followed your upgrade docs as you suggest to upgrade due to discontinuing of the support old Ubuntu version, since then this problem happens. And it just happened again, right now. Twice in today.

[mendoksai]

@nebulon Yes, here's the full timeline of changes:

1. Server was stable on Ubuntu 20.04 + kernel 5.4 for months
2. Upgraded to Ubuntu 22.04 + kernel 5.15 (following Cloudron upgrade docs) — instability started
3. Upgraded to Ubuntu 24.04 + kernel 6.8 (following Cloudron upgrade docs) — issue persists
4. Installed fail2ban and smartmontools via apt
5. No other custom modifications

All upgrades were done following the official Cloudron documentation. The crashes happen on both kernel 5.15 and 6.8, so it doesn't seem kernel-specific.

One thing that may be relevant: Docker is using cgroupfs driver with cgroup v2. The Cloudron systemd unit explicitly sets --exec-opt native.cgroupdriver=cgroupfs. Could there be a compatibility issue with Ubuntu 24.04's default cgroup v2?

The server just crashed again twice in one hour. Happy to provide SSH access if that would help debug this. This is urgent as my mail server runs on this machine.

[mendoksai]

Update: I renewed the expired domain and the app (Lychee) is now running properly. No containers in restart loop currently. The earlier crashes today were likely caused by the runner container still being in a stale state from before the domain renewal.

I have a cron job cleaning up zombie runners every 5 minutes, which seems to be working (log shows it removed 5 runners since setup).

Will monitor for the next few days and report back. If it stays stable, I'll mark this as resolved.

Thank you @girish @nebulon @joseph for your help!

[mendoksai]

@girish @nebulon Server crashed again last night. But this time the pattern is different — no containers in restart loop, no runner issues. The cron cleanup job is working. All containers were stable (Up 11 hours) before the crash.

The Docker journal shows the DNS resolver dying on its own:

23:38 - External DNS timeouts begin (185.12.64.2)
23:57 - Internal Docker DNS fails (172.18.0.1:53 i/o timeout)
23:59 - [resolver] connect failed: dial tcp 172.18.0.1:53: i/o timeout
00:xx - Server becomes unresponsive

There's also a container (different ID each time) producing "ignoring event" / "cleaning up dead shim" messages every minute — not sure if related.

This happens roughly at the same time every night (~23:00-00:00 UTC). All previous fixes applied (no restart loops, domain renewed, hardware clean). I'm running out of ideas on my end.

Would it be possible to get SSH-level support to debug this? I can provide access anytime. This is really urgent as it's been impacting my mail service daily for weeks now.

Thank you.