Server crashes caused by stopped app's runner container stuck in restart loop

[mendoksai]

Happened again. Every a few days.

[nebulon]

Do you by any chance have made some custom modifications to your ubuntu system like applying apt updates or so?

[mendoksai]

Yes, I followed your upgrade docs as you suggest to upgrade due to discontinuing of the support old Ubuntu version, since then this problem happens. And it just happened again, right now. Twice in today.

[mendoksai]

@nebulon Yes, here's the full timeline of changes:

1. Server was stable on Ubuntu 20.04 + kernel 5.4 for months
2. Upgraded to Ubuntu 22.04 + kernel 5.15 (following Cloudron upgrade docs) — instability started
3. Upgraded to Ubuntu 24.04 + kernel 6.8 (following Cloudron upgrade docs) — issue persists
4. Installed fail2ban and smartmontools via apt
5. No other custom modifications

All upgrades were done following the official Cloudron documentation. The crashes happen on both kernel 5.15 and 6.8, so it doesn't seem kernel-specific.

One thing that may be relevant: Docker is using cgroupfs driver with cgroup v2. The Cloudron systemd unit explicitly sets --exec-opt native.cgroupdriver=cgroupfs. Could there be a compatibility issue with Ubuntu 24.04's default cgroup v2?

The server just crashed again twice in one hour. Happy to provide SSH access if that would help debug this. This is urgent as my mail server runs on this machine.

[mendoksai]

Update: I renewed the expired domain and the app (Lychee) is now running properly. No containers in restart loop currently. The earlier crashes today were likely caused by the runner container still being in a stale state from before the domain renewal.

I have a cron job cleaning up zombie runners every 5 minutes, which seems to be working (log shows it removed 5 runners since setup).

Will monitor for the next few days and report back. If it stays stable, I'll mark this as resolved.

Thank you @girish @nebulon @joseph for your help!

[mendoksai]

@girish @nebulon Server crashed again last night. But this time the pattern is different — no containers in restart loop, no runner issues. The cron cleanup job is working. All containers were stable (Up 11 hours) before the crash.

The Docker journal shows the DNS resolver dying on its own:

23:38 - External DNS timeouts begin (185.12.64.2)
23:57 - Internal Docker DNS fails (172.18.0.1:53 i/o timeout)
23:59 - [resolver] connect failed: dial tcp 172.18.0.1:53: i/o timeout
00:xx - Server becomes unresponsive

There's also a container (different ID each time) producing "ignoring event" / "cleaning up dead shim" messages every minute — not sure if related.

This happens roughly at the same time every night (~23:00-00:00 UTC). All previous fixes applied (no restart loops, domain renewed, hardware clean). I'm running out of ideas on my end.

Would it be possible to get SSH-level support to debug this? I can provide access anytime. This is really urgent as it's been impacting my mail service daily for weeks now.

Thank you.

[joseph]

@mendoksai yes, write to me at support@cloudron.io . I can investigate.

[mendoksai]

Server was stable for 14 days after I fixed the DNS configuration myself. The original daily crash issue was resolved.

This morning I received Cloudron's security reboot email. Rebooted via dashboard. Server never came back. Ping responds, SSH returns kex_exchange_identification: Connection reset by peer. Hard reset via Hetzner Robot didn't help either.

So now I'm locked out of my own server because of an automatic security update that I didn't ask for and had no control over. My mail server is down, again.

I have to ask: is anyone actually testing these updates before pushing them? Every major issue I've had in the past two months has been triggered by an automatic update or upgrade. The previous instability started after a Cloudron update in February. Now this.

I need:

1. Help getting my server back online — I'll likely need to use Hetzner rescue mode
2. A way to permanently disable automatic security updates so I can apply them manually at a time that works for me
3. Some assurance that updates are being properly tested before being pushed to production servers

This is a production server running critical mail services. I can't keep being the QA tester for untested updates.

Are you guys vibe coding?

[james]

Hello @mendoksai

Sorry to read that you are having issues.
This sounds like something went wrong with the unattended security updates.

@mendoksai said:

because of an automatic security update that I didn't ask for and had no control over

This is default for every Cloudron installation.
Even if you don't ask for it now, you will ask for it if your server gets compromised due to apt packages not being updated with security updates.

---

When the server is running, even tho you can not connect, try to open the console from the hetzner dashboard and see what is written in the terminal.
First we need to understand what the issue is before tinkering with the system.

[mendoksai]

I used rescue mode and I saw the issue was with kernel and fixed the kernel version, disabled docker, Cloudron so I could connect. Then I did another investigation which seems Raid Controller had problem so I asked Hetzner checked they replaced Raid Controller and then things back to normal. So, I think the issue was more like happing from Raid Controller but just timing was with the upgrade and reboot gave the wrong impression, although I'm not 100% sure. Anyway, for now, it seems okay. You may close this ticket. Thank you.

[james]

Hello @mendoksai
That checks out with our other topic.
A RAID failure can cause such a catastrophic issue.