Server crashes caused by stopped app's runner container stuck in restart loop
-
A domain expired for one of my apps. I stopped the app via the Cloudron dashboard. However, the runner container remained in "Created" state and kept trying to join the network namespace of the stopped app container, causing cascading failures:
- Runner repeatedly fails with:
Cannot restart container <appid>-runner: cannot join network namespace of container: Container <id> is restarting, wait until the container is running - This eventually causes Docker DNS resolution failures (internal Docker DNS timeouts)
- Host MySQL becomes unreachable (
ECONNREFUSED 127.0.0.1:3306) - SSH stops accepting connections
- Server becomes completely unresponsive, requiring hard reboot
This has been happening daily for the past week.
What I did
- Stopped the app via Cloudron dashboard → runner remained in "Created" state
docker rm -f <appid>-runnerremoved the stuck runner- Main container shows "Exited (0)" and redis addon is still running — both untouched
Questions
- Will Cloudron's scheduler recreate the runner container for a stopped app? If so, how do I prevent this?
- Is there a proper way to fully stop an app including its runner when the domain has expired?
- Should I also stop the redis addon container for this app?
Relevant box.log pattern (repeating every 15-60 min):
box:scheduler could not run task runner: (HTTP code 500) server error - Cannot restart container <appid>-runner: cannot join network namespace of containerAlso seeing on every boot:
Error: listen EADDRNOTAVAIL: address not available 172.18.0.1:3003cloudron-support --troubleshoot Vendor: System manufacturer Product: System Product Name Linux: 6.8.0-106-generic Ubuntu: noble 24.04 Execution environment: none none Processor: Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz BIOS Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz To Be Filled By O.E.M. CPU @ 3.4GHz x 8 RAM: 32796076KB Disk: /dev/sda3 909G [OK] node version is correct [OK] IPv6 is enabled and public IPv6 address is working [OK] docker is running [OK] docker version is correct [OK] MySQL is running [OK] netplan is good [OK] DNS is resolving via systemd-resolved [OK] unbound is running [OK] nginx is running [OK] dashboard cert is valid [OK] dashboard is reachable via loopback [FAIL] Database migrations are pending. Last migration in DB: /20260217120000-mailPasswords-create-table.js. Last migration file: /package.json. Please run 'cloudron-support --apply-db-migrations' to apply the migrations. [OK] Service 'mysql' is running and healthy [OK] Service 'postgresql' is running and healthy [OK] Service 'mongodb' is running and healthy [OK] Service 'mail' is running and healthy [OK] Service 'graphite' is running and healthy [OK] Service 'sftp' is running and healthy [OK] box v9.1.3 is running [OK] Dashboard is reachable via domain name [OK] Domain is valid and has not expired - Runner repeatedly fails with:
-
Update: Confirmed that Cloudron recreates the runner container on every boot, even though the app is stopped via the dashboard.
After each reboot:
- Main container:
Exited (0)✓ - Runner container:
Created← this is the problem - Redis addon:
Up← also still running
The runner in "Created" state triggers the scheduler loop → "cannot join network namespace" errors every 15-60 min → eventually cascading into Docker DNS failure → MySQL unreachable → full server lockup.
I've been manually removing the runner with
docker rm -f <appid>-runnerafter each reboot, but this is not sustainable.Is there a way to prevent the scheduler from recreating the runner for a stopped app? Or should I uninstall the app entirely to stop this cycle? The app's domain has expired but I'd like to keep the data for when I renew it.
Thanks for any guidance.
- Main container:
-
@mendoksai the container getting created is not a problem. The container is created but not run for stopped apps (i.e docker container create vs docker container run). The issue is also not related to domains (and it's expiry).
I haven't been able to reproduce this issue though.
I think the issue is actually that box code is unable to control docker. Or maybe docker is not running commands properly. For example,
Container <id> is restarting, wait until the container is running. This already indicates the stopped app is in incorrect state. The Container has to be in stopped state. Are there any errors injournalctl -u docker -fa? The rest of the errors like redis not stopping, cron container error is all the same issue of docker not running containers properly.Error: listen EADDRNOTAVAIL: address not available 172.18.0.1:3003is similar. Docker is supposed to create thecloudronnetwork in that IP. Can't see how it can be unavailable.Can you give more information on your environment? Are other apps running properly?
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login