Upgrade to Cloudron 9.0.7 stuck on "Removing containers for upgrade" + "Configuring"
-
Hi!
Yesterday the update notification popped up for me and I thought: why not do a major upgrade on a Friday afternoon, what could possible go wrong
Joke aside:
The update seemed to progress well. Server restarted and event log shows 9.0.7 as successful.
But none of the apps work. Every single one shows "Configuring" in the app overview. The individual logs show nothing out of the ordinary and restarting individual apps does nothing.
Also in the main app overview Cloudron shows the notification "Removing containers for upgrade"
Tried rebooting the server already twice.
Last few lines of the upgrade logs:
7f1cd0eeb2e8: Already exists 88ca0e28f7e0: Already exists a7805ce42b2e: Already exists 758b3fac755b: Already exists 11511e698c34: Already exists 15ce653c2527: Pulling fs layer 15ce653c2527: Verifying Checksum 15ce653c2527: Download complete 15ce653c2527: Pull complete Digest: sha256:9f3609969a5757837505c584c98246a3035a84a273b9be491665ac026423fd5f Status: Downloaded newer image for registry.ipv4.docker.com/cloudron/turn@sha256:9f3609969a5757837505c584c98246a3035a84a273b9be491665ac026423fd5f registry.ipv4.docker.com/cloudron/turn:1.8.2@sha256:9f3609969a5757837505c584c98246a3035a84a273b9be491665ac026423fd5f Untagged: registry.ipv4.docker.com/cloudron/turn:1.8.2@sha256:9f3609969a5757837505c584c98246a3035a84a273b9be491665ac026423fd5f 2025-11-07T23:29:56 ==> installer: stop box service for update 2025-11-07T23:30:01 ==> installer: switching the box code 2025-11-07T23:30:02 ==> installer: calling box setup script 2025-11-07T23:30:02 ==> start: Cloudron Start media:x:500: 2025-11-07T23:30:02 ==> start: Configuring docker Synchronizing state of apparmor.service with SysV service script with /usr/lib/systemd/systemd-sysv-install. Executing: /usr/lib/systemd/systemd-sysv-install enable apparmor 2025-11-07T23:30:04 ==> start: Adding userland-proxy=false to docker 2025-11-07T23:30:20 ==> start: Ensuring directories 2025-11-07T23:30:20 ==> start: Configuring journald 2025-11-07T23:30:21 ==> start: Setting up unbound 2025-11-07T23:30:21 ==> start: Adding systemd services Synchronizing state of unbound.service with SysV service script with /usr/lib/systemd/systemd-sysv-install. Executing: /usr/lib/systemd/systemd-sysv-install enable unbound Synchronizing state of cron.service with SysV service script with /usr/lib/systemd/systemd-sysv-install. Executing: /usr/lib/systemd/systemd-sysv-install enable cron Synchronizing state of rpcbind.service with SysV service script with /usr/lib/systemd/systemd-sysv-install. Executing: /usr/lib/systemd/systemd-sysv-install disable rpcbind 2025-11-07T23:30:30 ==> start: Configuring sudoers 2025-11-07T23:30:30 ==> start: Unconfiguring collectd Synchronizing state of collectd.service with SysV service script with /usr/lib/systemd/systemd-sysv-install. Executing: /usr/lib/systemd/systemd-sysv-install disable collectd Removed "/etc/systemd/system/multi-user.target.wants/collectd.service". 2025-11-07T23:30:32 ==> start: Configuring logrotate 2025-11-07T23:30:32 ==> start: Adding motd message for admins 2025-11-07T23:30:32 ==> start: Configuring nginx 2025-11-07T23:30:33 ==> start: Starting mysql mysqladmin: [Warning] Using a password on the command line interface can be insecure. Warning: Since password will be sent to server in plain text, use ssl connection to ensure password safety. mysql: [Warning] Using a password on the command line interface can be insecure. mysql: [Warning] Using a password on the command line interface can be insecure. 2025-11-07T23:30:33 ==> start: Migrating data [INFO] Processed migration 20250307100449-tokens-add-allowedIpRangesJson [INFO] Processed migration 20250507122501-dockerRegistries-create-table [INFO] Processed migration 20250608103424-users-make-avatar-nullable [INFO] Processed migration 20250617133955-tasks-add-pending [INFO] Processed migration 20250618184304-volumes-truncate-hostPath Changing database charset Changing charset of userGroups Changing charset of users Changing charset of domains Changing charset of tasks Changing charset of volumes Changing charset of apps Changing charset of backups Changing charset of mail Changing charset of appAddonConfigs Changing charset of appEnvVars Changing charset of appMounts Changing charset of appPasswords Changing charset of appPortBindings Changing charset of applinks Changing charset of archives Changing charset of blobs Changing charset of dockerRegistries Changing charset of eventlog Changing charset of groupMembers Changing charset of locations Changing charset of locks Changing charset of mailboxes Changing charset of migrations Changing charset of notifications Changing charset of oidcClients Changing charset of settings Changing charset of tokens [INFO] Processed migration 20250618200400-fix-charset-collate [INFO] Processed migration 20250626130733-apps-add-updateInfoJson [INFO] Processed migration 20250713133718-oidcClients-separate-ids-for-oidc-proxyauth [INFO] Processed migration 20250716070056-fixup-linode-objectstorage-settings [INFO] Processed migration 20250716130216-tasks-add-completed [INFO] Processed migration 20250718122114-tasks-alter-type Moving existing cache and snapshot file into /home/yellowtent/platformdata/backup/3ff6f679-929d-4bc4-b8aa-969bd7ba18cb Delete any existing rsync cache files since old one has no integrity information [INFO] Processed migration 20250724102340-backupSites-create-table Backup counts. rsync: 0 tgz: 255 total: 255 . theOneFormat: tgz Adjusting remotePath of existing tgz backups with .tar.gz [INFO] Processed migration 20250724141339-backups-add-siteId [INFO] Processed migration 20250812102445-backups-add-integrityJson [INFO] Processed migration 20250812141107-backups-add-statsJson [INFO] Done 2025-11-07T23:30:40 ==> start: Changing ownership 2025-11-07T23:30:40 ==> start: Starting cloudron-syslog 2025-11-07T23:30:40 ==> start: Starting Cloudron 2025-11-07T23:30:42 ==> start: Almost doneHow can I go about debugging this?
-
Hi!
Yesterday the update notification popped up for me and I thought: why not do a major upgrade on a Friday afternoon, what could possible go wrong
Joke aside:
The update seemed to progress well. Server restarted and event log shows 9.0.7 as successful.
But none of the apps work. Every single one shows "Configuring" in the app overview. The individual logs show nothing out of the ordinary and restarting individual apps does nothing.
Also in the main app overview Cloudron shows the notification "Removing containers for upgrade"
Tried rebooting the server already twice.
Last few lines of the upgrade logs:
7f1cd0eeb2e8: Already exists 88ca0e28f7e0: Already exists a7805ce42b2e: Already exists 758b3fac755b: Already exists 11511e698c34: Already exists 15ce653c2527: Pulling fs layer 15ce653c2527: Verifying Checksum 15ce653c2527: Download complete 15ce653c2527: Pull complete Digest: sha256:9f3609969a5757837505c584c98246a3035a84a273b9be491665ac026423fd5f Status: Downloaded newer image for registry.ipv4.docker.com/cloudron/turn@sha256:9f3609969a5757837505c584c98246a3035a84a273b9be491665ac026423fd5f registry.ipv4.docker.com/cloudron/turn:1.8.2@sha256:9f3609969a5757837505c584c98246a3035a84a273b9be491665ac026423fd5f Untagged: registry.ipv4.docker.com/cloudron/turn:1.8.2@sha256:9f3609969a5757837505c584c98246a3035a84a273b9be491665ac026423fd5f 2025-11-07T23:29:56 ==> installer: stop box service for update 2025-11-07T23:30:01 ==> installer: switching the box code 2025-11-07T23:30:02 ==> installer: calling box setup script 2025-11-07T23:30:02 ==> start: Cloudron Start media:x:500: 2025-11-07T23:30:02 ==> start: Configuring docker Synchronizing state of apparmor.service with SysV service script with /usr/lib/systemd/systemd-sysv-install. Executing: /usr/lib/systemd/systemd-sysv-install enable apparmor 2025-11-07T23:30:04 ==> start: Adding userland-proxy=false to docker 2025-11-07T23:30:20 ==> start: Ensuring directories 2025-11-07T23:30:20 ==> start: Configuring journald 2025-11-07T23:30:21 ==> start: Setting up unbound 2025-11-07T23:30:21 ==> start: Adding systemd services Synchronizing state of unbound.service with SysV service script with /usr/lib/systemd/systemd-sysv-install. Executing: /usr/lib/systemd/systemd-sysv-install enable unbound Synchronizing state of cron.service with SysV service script with /usr/lib/systemd/systemd-sysv-install. Executing: /usr/lib/systemd/systemd-sysv-install enable cron Synchronizing state of rpcbind.service with SysV service script with /usr/lib/systemd/systemd-sysv-install. Executing: /usr/lib/systemd/systemd-sysv-install disable rpcbind 2025-11-07T23:30:30 ==> start: Configuring sudoers 2025-11-07T23:30:30 ==> start: Unconfiguring collectd Synchronizing state of collectd.service with SysV service script with /usr/lib/systemd/systemd-sysv-install. Executing: /usr/lib/systemd/systemd-sysv-install disable collectd Removed "/etc/systemd/system/multi-user.target.wants/collectd.service". 2025-11-07T23:30:32 ==> start: Configuring logrotate 2025-11-07T23:30:32 ==> start: Adding motd message for admins 2025-11-07T23:30:32 ==> start: Configuring nginx 2025-11-07T23:30:33 ==> start: Starting mysql mysqladmin: [Warning] Using a password on the command line interface can be insecure. Warning: Since password will be sent to server in plain text, use ssl connection to ensure password safety. mysql: [Warning] Using a password on the command line interface can be insecure. mysql: [Warning] Using a password on the command line interface can be insecure. 2025-11-07T23:30:33 ==> start: Migrating data [INFO] Processed migration 20250307100449-tokens-add-allowedIpRangesJson [INFO] Processed migration 20250507122501-dockerRegistries-create-table [INFO] Processed migration 20250608103424-users-make-avatar-nullable [INFO] Processed migration 20250617133955-tasks-add-pending [INFO] Processed migration 20250618184304-volumes-truncate-hostPath Changing database charset Changing charset of userGroups Changing charset of users Changing charset of domains Changing charset of tasks Changing charset of volumes Changing charset of apps Changing charset of backups Changing charset of mail Changing charset of appAddonConfigs Changing charset of appEnvVars Changing charset of appMounts Changing charset of appPasswords Changing charset of appPortBindings Changing charset of applinks Changing charset of archives Changing charset of blobs Changing charset of dockerRegistries Changing charset of eventlog Changing charset of groupMembers Changing charset of locations Changing charset of locks Changing charset of mailboxes Changing charset of migrations Changing charset of notifications Changing charset of oidcClients Changing charset of settings Changing charset of tokens [INFO] Processed migration 20250618200400-fix-charset-collate [INFO] Processed migration 20250626130733-apps-add-updateInfoJson [INFO] Processed migration 20250713133718-oidcClients-separate-ids-for-oidc-proxyauth [INFO] Processed migration 20250716070056-fixup-linode-objectstorage-settings [INFO] Processed migration 20250716130216-tasks-add-completed [INFO] Processed migration 20250718122114-tasks-alter-type Moving existing cache and snapshot file into /home/yellowtent/platformdata/backup/3ff6f679-929d-4bc4-b8aa-969bd7ba18cb Delete any existing rsync cache files since old one has no integrity information [INFO] Processed migration 20250724102340-backupSites-create-table Backup counts. rsync: 0 tgz: 255 total: 255 . theOneFormat: tgz Adjusting remotePath of existing tgz backups with .tar.gz [INFO] Processed migration 20250724141339-backups-add-siteId [INFO] Processed migration 20250812102445-backups-add-integrityJson [INFO] Processed migration 20250812141107-backups-add-statsJson [INFO] Done 2025-11-07T23:30:40 ==> start: Changing ownership 2025-11-07T23:30:40 ==> start: Starting cloudron-syslog 2025-11-07T23:30:40 ==> start: Starting Cloudron 2025-11-07T23:30:42 ==> start: Almost doneHow can I go about debugging this?
@perelin said in Upgrade to Cloudron 9.0.7 stuck on "Removing containers for upgrade" + "Configuring":
why not do a major upgrade on a Friday afternoon
Nothing beats shipping to production on friday afternoon!
Already tried
cloudron-support --troubleshoot, rebooting the server?
