Cloudron 10.1.5 update failed – Database migrations pending, firewall failure and Docker networking broken

[weesaal]

Hello Cloudron Team,

I'm experiencing critical issues with my Cloudron installation following an automatic update. The Cloudron dashboard is no longer allowing me to log in, the firewall service is failing, and Docker containers cannot establish outbound internet connections.

I've performed extensive troubleshooting and would appreciate guidance on safely restoring the server without losing application data.

1. Server information

• Cloudron: 10.1.5
• OS: Ubuntu 24.04 LTS
• Kernel: 6.8.0-147-generic
• Docker: 29.6.2
• Docker firewall backend: iptables
• Hosting provider: Contabo (KVM VPS)
• Dashboard: https://my.baskethunt.com

Docker 29.6.2 was installed on September 13, 2026, and kernel 6.8.0-147 was installed on October 14, 2026.

2. Cloudron dashboard login failure

After the automatic update, the login page loads successfully, but submitting my credentials produces:

Internal error, try again later

The Cloudron box log contains:

HttpError: Unknown column 'inviteTokenCreationTime' in 'field list'
SQL Error: ER_BAD_FIELD_ERROR (1054)

Running cloudron-support --troubleshoot shows:

PASS: 32
WARN: 0
FAIL: 1
SKIP: 0

[FAIL] Database migrations are pending.

Last migration in DB:
20260908124000-appPasswords-add-ownerId.js

Last migration file:
20261009112100-oidcClients-add-allowTokenExchange.js

I attempted:

sudo cloudron-support --apply-db-migrations

However, the process failed while starting the firewall service:

Job for cloudron-firewall.service failed
[FAIL] Database migrations failed

3. Cloudron firewall service failure

Running systemctl status cloudron-firewall.service produces:

iptables: Bad rule (does a matching rule exist in that chain?).
ip6tables: Bad rule (does a matching rule exist in that chain?).
iptables: No chain/target/match by that name.

cloudron-firewall.service:
Failed with result 'exit-code'

The firewall startup script attempts to insert rules into DOCKER-USER, but this chain appears to be unavailable when Cloudron tries to use it.

4. Missing Docker firewall and NAT rules

I checked Docker's networking configuration.

Docker reports:

Firewall Backend: iptables

However:

sudo iptables -t nat -S

Returns only:

-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT

No Docker NAT/MASQUERADE rules are present.

Additionally, iptables-legacy NAT and filtering checks showed no Docker-managed rules.

I restarted Docker using:

sudo systemctl restart docker

Immediately afterward, the DOCKER-USER chain appeared:

-N DOCKER-USER

But subsequent checks showed that the chain was missing again.

Restarting cloudron-firewall.service still fails with the same error.

I have not manually flushed the firewall, switched iptables backends, or edited Docker configuration files.

5. Docker container internet connectivity failure

My Chatwoot installation is also affected.

Outgoing WhatsApp messages, including approved templates, remain stuck on "Sending."

Chatwoot's Sidekiq jobs report:

SendReplyJob
Channels::Whatsapp::TemplatesSyncJob

Net::OpenTimeout:
Failed to open TCP connection to
graph.facebook.com:443 (execution expired)

Connectivity tests show:

• The Ubuntu host successfully reaches Google and Meta's Graph API.
• The Chatwoot container times out connecting to both.
• IPv4-only connections also time out inside Chatwoot.
• IPv4 forwarding is enabled.
• The br_netfilter module is loaded.
• Both bridge-nf-call-iptables and bridge-nf-call-ip6tables are enabled.

This appears to be a broader Docker container networking issue rather than a WhatsApp configuration issue.

6. Additional checks completed

I've also verified that:

• box.service is running.
• Docker is running.
• UFW is inactive.
• nftables.service is disabled and inactive.
• netfilter-persistent.service is not installed.
• The Docker cloudron bridge network is not marked internal.
• /etc/docker/daemon.json does not exist.
• Docker's systemd Cloudron override is present.
• The Docker logs show successful daemon initialization.

7. Assistance requested

Could you please help determine:

1. Why Docker's firewall and NAT rules are missing or disappearing?
2. Why cloudron-firewall.service cannot initialize?
3. Whether the current Docker and Cloudron versions have a known compatibility issue?
4. What is the recommended recovery procedure to complete the pending database migrations and restore dashboard login?
5. How can we restore outbound connectivity to Cloudron-hosted applications?

This is an active server, so preserving application data and minimizing downtime are important.

I can provide additional logs, screenshots, and diagnostic information if required.

Thank you for your assistance.

[james]

Hello @weesaal and welcome to the Cloudron community forum

I think you might be running into the same or similar issue as here https://forum.cloudron.io/topic/14652/could-not-load-dashboard-website-with-loopback-check-boxerror-unknown-column-pending-in-field-list-update-from-8.3.2-to-9.0.13

the solution was:

@joseph said:

Please run this manually and it should start again:

/home/yellowtent/box/setup/start.sh

Although cloudron-support --apply-db-migrations does run /home/yellowtent/box/setup/start.sh.

You did run cloudron-support with sudo, which might not work correctly.
Just to make sure, please run cloudron-support as the root user:

sudo su -
cloudron-support --apply-db-migrations

and report back, if that resolves the issue.

[weesaal]

Hi @james,

Thank you for your response and for sharing the related discussion.

Unfortunately, I've already tried the suggested recovery procedure, but the issue remains unresolved.

1. Database migration recovery already attempted

I previously ran the following command as the root user:

sudo cloudron-support --apply-db-migrations

The process started but failed when attempting to initialize Cloudron's firewall:

Job for cloudron-firewall.service failed because the control process exited with error code.
[FAIL] Database migrations failed

I also verified that root has working sudo permissions:

sudo -n true && echo "ROOT SUDO WORKS" || echo "ROOT SUDO FAILED"

Result:

ROOT SUDO WORKS

Therefore, the sudo permission issue mentioned in the linked discussion doesn't appear to be the cause in my case.

I haven't separately executed /home/yellowtent/box/setup/start.sh, since the migration recovery command already invokes that startup process and stops at the firewall error.

2. Cloudron firewall failure

The cloudron-firewall.service repeatedly fails with:

iptables: Bad rule (does a matching rule exist in that chain?).
ip6tables: Bad rule (does a matching rule exist in that chain?).
iptables: No chain/target/match by that name.

The firewall initialization script attempts to insert a rule into the DOCKER-USER chain, but that chain appears to be missing or unavailable.

3. Docker networking troubleshooting completed

I have checked the following:

• Docker 29.6.2 is running, with the iptables firewall backend.
• IPv4 forwarding is enabled.
• The br_netfilter module is loaded.
• Both IPv4 and IPv6 bridge netfilter settings are enabled.
• UFW is inactive.
• The nftables service is inactive and disabled.
• No netfilter-persistent service is installed.
• No explicit Docker firewall configuration exists in /etc/docker/daemon.json.
• The Cloudron Docker network uses the bridge driver and is not marked internal.
• Docker's NAT/MASQUERADE rules are absent from both iptables backends.
• Docker's standard forwarding/filtering chains are also missing.

I have also restarted Docker:

sudo systemctl restart docker

Immediately afterward, the DOCKER-USER chain appeared, but subsequent checks showed that it was missing again.

Restarting the Cloudron firewall service did not resolve the issue:

sudo systemctl restart cloudron-firewall.service

The same error persists.

4. Cloudron login is still broken

The login page loads, but attempting to sign in results in:

Internal error, try again later

The box log reports:

Unknown column 'inviteTokenCreationTime' in 'field list'
ER_BAD_FIELD_ERROR (1054)

The diagnostic tool reports pending database migrations, while the other 32 checks passed.

5. Application networking is also affected

My Chatwoot application cannot establish outbound HTTPS connections to Meta or Google from inside its container, although the Ubuntu host can reach both.

Chatwoot's Sidekiq logs repeatedly show:

Net::OpenTimeout:
Failed to open TCP connection to graph.facebook.com:443

As a result, approved WhatsApp templates remain stuck on "Sending."

This suggests that the Docker networking and Cloudron firewall issues may be related.

6. Urgent concern about application data

I'm now particularly concerned about the safety of my application data.

One of the applications hosted on this Cloudron server has not been backed up for approximately one week, and it contains important data that I cannot afford to lose.

The application data may still be intact, but with the Cloudron dashboard inaccessible and the platform update incomplete, I'm concerned that further recovery attempts could put this data at risk.

Before proceeding with additional repairs, could you please advise on the safest way to preserve the existing application data and database in their current state?

Specifically, I would appreciate guidance on whether we should create a server-level snapshot or perform an application/database backup through SSH before attempting any further recovery.

I would also appreciate your help with the following:

1. Identifying why the Docker-managed firewall chains and NAT rules are missing.
2. Restoring cloudron-firewall.service without disrupting existing application data.
3. Completing the pending Cloudron database migrations safely.
4. Restoring the Cloudron dashboard and application connectivity.

I've already spent considerable time diagnosing this issue, and I'm hesitant to perform further restarts, upgrades, or firewall changes without guidance because of the risk to my application data.

Could a Cloudron engineer please review this and recommend the safest recovery procedure?

Thank you for your continued assistance.

[joseph]

@weesaal I replied from support earlier today but I see you have responded now. I am looking into this.

[joseph]

So, on this server legacy iptable modules are being loaded in the kernel. Maybe some change in your server provider? Cloudron doesn't configure or load these modules (cloudron uses nft backend for iptables). Docker is not configuring iptables properly because of this.

root@vmi3573607:~# lsmod | grep -E 'iptable_|ip6table_'
ip6table_filter        12288  0
ip6_tables             36864  1 ip6table_filter
iptable_filter         12288  0
iptable_nat            12288  0
nf_nat                 61440  5 xt_nat,nft_chain_nat,iptable_nat,xt_MASQUERADE,xt_REDIRECT
ip_tables              32768  2 iptable_filter,iptable_nat
x_tables               65536  18 ip6table_filter,xt_conntrack,iptable_filter,nft_compat,xt_LOG,xt_multiport,xt_tcpudp,xt_hashlimit,xt_addrtype,xt_recent,xt_nat,xt_set,ip6_tables,ip_tables,iptable_nat,xt_limit,xt_MASQUERADE,xt_REDIRECT
root@vmi3573607:~# cat /proc/net/ip_tables_names /proc/net/ip6_tables_names
filter
nat
filter

I manually unloaded them and it work again:

sudo modprobe -r iptable_nat iptable_filter ip6table_filter

Could an agent or something have loaded these legacy modules?