Cloudron update recreated docker cloudron network causing database connectivity issues for all apps

[leighmhart]

Hi team,

After upgrading to 9.1.7 and a few days later responding to the "reboot required" reminder email, none of my apps would come back up properly - some reported database connection errors so after staring at the logs for a while I set Claude loose on my environment to try and figure out what was going on.

After about an hour of token burning I believe the root cause was:

Root cause: Cloudron's update recreated the Docker cloudron network, assigning a new bridge ID (br-redacted). The nftables ip raw PREROUTING chain had stale security rules from the OLD bridge ID (br-also-redacted). These stale rules appeared BEFORE the new bridge's rules and dropped all container-to-database packets at the raw level — BEFORE the FORWARD chain was even reached.

Why: Cloudron adds new bridge rules to ip raw PREROUTING when the network is created but does NOT remove old rules for the previous bridge ID. Old rules silently drop all traffic from containers on the new bridge to database IPs.

How to diagnose:

1. Check sudo nft list table ip raw for rules with old bridge IDs that DROP container IPs
2. Look for high counters on iifname != "br-XXXXXXXX" rules — that's the stale rule doing the dropping
3. The iptables FORWARD chain counter for the database IP will show 0 (packets never reach FORWARD)
4. tcpdump on the veth shows SYNs leaving the container, but no SYN-ACKs return

Fix:

sudo nft -a list chain ip raw PREROUTING | grep br-<OLD-ID>  # find stale rule handles
sudo nft delete rule ip raw PREROUTING handle <N>             # delete each stale rule

Delete all rules referencing the old bridge ID. The new bridge rules (for the current bridge) remain and are correct.

How to apply: After any Cloudron update that changes the Docker network bridge ID, check nft list table ip raw for stale bridge rules. Also check after any reboot where Docker recreation of the cloudron network is suspected.

Hopefully this helps anyone else who encounters this issue - it may be isolated to me, I don't know, but I'm really impressed that Claude was able to figure this out.

Logs: I did not capture the cloudron-support --troubleshoot for this case as I've resolved the issue, but can do if the dev team thinks this will help them figure out what went wrong.

[james]

Hello @leighmhart and welcome to the Cloudron forum

Thanks for the detailed report.
We will have to look into this.