Upgrade to Cloudron 9.0.7 stuck on "Removing containers for upgrade" + "Configuring"

[perelin]

Hi!

Yesterday the update notification popped up for me and I thought: why not do a major upgrade on a Friday afternoon, what could possible go wrong

Joke aside:

The update seemed to progress well. Server restarted and event log shows 9.0.7 as successful.

But none of the apps work. Every single one shows "Configuring" in the app overview. The individual logs show nothing out of the ordinary and restarting individual apps does nothing.

Also in the main app overview Cloudron shows the notification "Removing containers for upgrade"

Tried rebooting the server already twice.

Last few lines of the upgrade logs:

7f1cd0eeb2e8: Already exists
88ca0e28f7e0: Already exists
a7805ce42b2e: Already exists
758b3fac755b: Already exists
11511e698c34: Already exists
15ce653c2527: Pulling fs layer
15ce653c2527: Verifying Checksum
15ce653c2527: Download complete
15ce653c2527: Pull complete
Digest: sha256:9f3609969a5757837505c584c98246a3035a84a273b9be491665ac026423fd5f
Status: Downloaded newer image for registry.ipv4.docker.com/cloudron/turn@sha256:9f3609969a5757837505c584c98246a3035a84a273b9be491665ac026423fd5f
registry.ipv4.docker.com/cloudron/turn:1.8.2@sha256:9f3609969a5757837505c584c98246a3035a84a273b9be491665ac026423fd5f
Untagged: registry.ipv4.docker.com/cloudron/turn:1.8.2@sha256:9f3609969a5757837505c584c98246a3035a84a273b9be491665ac026423fd5f
2025-11-07T23:29:56 ==> installer: stop box service for update
2025-11-07T23:30:01 ==> installer: switching the box code
2025-11-07T23:30:02 ==> installer: calling box setup script
2025-11-07T23:30:02 ==> start: Cloudron Start
media:x:500:
2025-11-07T23:30:02 ==> start: Configuring docker
Synchronizing state of apparmor.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable apparmor
2025-11-07T23:30:04 ==> start: Adding userland-proxy=false to docker
2025-11-07T23:30:20 ==> start: Ensuring directories
2025-11-07T23:30:20 ==> start: Configuring journald
2025-11-07T23:30:21 ==> start: Setting up unbound
2025-11-07T23:30:21 ==> start: Adding systemd services
Synchronizing state of unbound.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable unbound
Synchronizing state of cron.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable cron
Synchronizing state of rpcbind.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install disable rpcbind
2025-11-07T23:30:30 ==> start: Configuring sudoers
2025-11-07T23:30:30 ==> start: Unconfiguring collectd
Synchronizing state of collectd.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install disable collectd
Removed "/etc/systemd/system/multi-user.target.wants/collectd.service".
2025-11-07T23:30:32 ==> start: Configuring logrotate
2025-11-07T23:30:32 ==> start: Adding motd message for admins
2025-11-07T23:30:32 ==> start: Configuring nginx
2025-11-07T23:30:33 ==> start: Starting mysql
mysqladmin: [Warning] Using a password on the command line interface can be insecure.
Warning: Since password will be sent to server in plain text, use ssl connection to ensure password safety.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
2025-11-07T23:30:33 ==> start: Migrating data
[INFO] Processed migration 20250307100449-tokens-add-allowedIpRangesJson
[INFO] Processed migration 20250507122501-dockerRegistries-create-table
[INFO] Processed migration 20250608103424-users-make-avatar-nullable
[INFO] Processed migration 20250617133955-tasks-add-pending
[INFO] Processed migration 20250618184304-volumes-truncate-hostPath
Changing database charset
Changing charset of userGroups
Changing charset of users
Changing charset of domains
Changing charset of tasks
Changing charset of volumes
Changing charset of apps
Changing charset of backups
Changing charset of mail
Changing charset of appAddonConfigs
Changing charset of appEnvVars
Changing charset of appMounts
Changing charset of appPasswords
Changing charset of appPortBindings
Changing charset of applinks
Changing charset of archives
Changing charset of blobs
Changing charset of dockerRegistries
Changing charset of eventlog
Changing charset of groupMembers
Changing charset of locations
Changing charset of locks
Changing charset of mailboxes
Changing charset of migrations
Changing charset of notifications
Changing charset of oidcClients
Changing charset of settings
Changing charset of tokens
[INFO] Processed migration 20250618200400-fix-charset-collate
[INFO] Processed migration 20250626130733-apps-add-updateInfoJson
[INFO] Processed migration 20250713133718-oidcClients-separate-ids-for-oidc-proxyauth
[INFO] Processed migration 20250716070056-fixup-linode-objectstorage-settings
[INFO] Processed migration 20250716130216-tasks-add-completed
[INFO] Processed migration 20250718122114-tasks-alter-type
Moving existing cache and snapshot file into /home/yellowtent/platformdata/backup/3ff6f679-929d-4bc4-b8aa-969bd7ba18cb
Delete any existing rsync cache files since old one has no integrity information
[INFO] Processed migration 20250724102340-backupSites-create-table
Backup counts. rsync: 0 tgz: 255 total: 255 . theOneFormat: tgz
Adjusting remotePath of existing tgz backups with .tar.gz
[INFO] Processed migration 20250724141339-backups-add-siteId
[INFO] Processed migration 20250812102445-backups-add-integrityJson
[INFO] Processed migration 20250812141107-backups-add-statsJson
[INFO] Done
2025-11-07T23:30:40 ==> start: Changing ownership
2025-11-07T23:30:40 ==> start: Starting cloudron-syslog
2025-11-07T23:30:40 ==> start: Starting Cloudron
2025-11-07T23:30:42 ==> start: Almost done

How can I go about debugging this?

[andreasdueren]

@perelin said in Upgrade to Cloudron 9.0.7 stuck on "Removing containers for upgrade" + "Configuring":

why not do a major upgrade on a Friday afternoon

Nothing beats shipping to production on friday afternoon!

Already tried cloudron-support --troubleshoot, rebooting the server?

[joseph]

The update itself succeeded. The docker/containers are stuck. Usually, a reboot of the server makes docker get unstuck

[perelin]

Thanks for the answers. Yes, sorry forgot to mention: already rebooted the server. Just tried it again. Containers keep being stuck.

[joseph]

@perelin can you write to support@cloudron.io? we can look into why it's getting stuck .

also, maybe the logs at /home/yellowtent/platformdata/logs/box.log has something ?

[perelin]

@joseph Thanks! I wrote to support@ and will document the findings here.

box.log reveals this:

2025-11-08T06:26:02.896Z box:server ==========================================
2025-11-08T06:26:02.897Z box:server            Cloudron 9.0.7  
2025-11-08T06:26:02.897Z box:server ==========================================
2025-11-08T06:26:02.897Z box:platform initialize: start platform
2025-11-08T06:26:02.899Z box:tasks stopAllTasks: 0 tasks are running. sending abort signal
2025-11-08T06:26:02.899Z box:shell tasks: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/stoptask.sh all
2025-11-08T06:26:03.022Z box:locks releaseAll: all locks released
2025-11-08T06:26:03.030Z box:reverseproxy writeDashboardConfig: writing dashboard config for p2lab.com
2025-11-08T06:26:03.055Z box:shell reverseproxy: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/restartservice.sh nginx
2025-11-08T06:26:03.617Z box:platform onActivated: starting post activation services
2025-11-08T06:26:03.618Z box:platform startInfra: checking infrastructure
2025-11-08T06:26:03.619Z box:platform startInfra: updating infrastructure from 49.8.0 to 49.9.0
2025-11-08T06:26:03.619Z box:platform markApps: reconfiguring apps
2025-11-08T06:26:03.619Z box:reverseproxy removeAppConfigs: removing app nginx configs
2025-11-08T06:26:03.683Z box:platform removeAllContainers: removing all containers for infra upgrade
2025-11-08T06:26:03.683Z box:shell platform: docker ps -qa --filter label=isCloudronManaged
2025-11-08T06:26:03.823Z box:platform createDockerNetwork: recreating docker network
2025-11-08T06:26:03.823Z box:shell platform: docker network rm -f cloudron
2025-11-08T06:26:03.857Z box:shell platform: docker network rm -f cloudron errored BoxError: docker exited with code 1 signal null
    at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:82:23)
    at ChildProcess.emit (node:events:519:28)
    at maybeClose (node:internal/child_process:1101:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5) {
  reason: 'Shell Error',
  details: {},
  stdout: <Buffer >,
  stdoutLineCount: 0,
  stderr: <Buffer 45 72 72 6f 72 20 72 65 73 70 6f 6e 73 65 20 66 72 6f 6d 20 64 61 65 6d 6f 6e 3a 20 65 72 72 6f 72 20 77 68 69 6c 65 20 72 65 6d 6f 76 69 6e 67 20 6e ... 71 more bytes>,
  stderrLineCount: 2,
  code: 1,
  signal: null,
  timedOut: false,
  terminated: false
}
2025-11-08T06:26:03.861Z box:platform startInfra: Failed to start services. retry=false (attempt 0): docker exited with code 1 signal null
2025-11-08T06:26:03.861Z box:platform BoxError: docker exited with code 1 signal null
    at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:82:23)
    at ChildProcess.emit (node:events:519:28)
    at maybeClose (node:internal/child_process:1101:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)
2025-11-08T06:33:21.776Z box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name
2025-11-08T06:33:21.797Z box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name
2025-11-08T06:33:21.802Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target
2025-11-08T06:33:21.885Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/platformdata
2025-11-08T06:33:21.904Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/boxdata/box
2025-11-08T06:33:21.911Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/boxdata/mail
2025-11-08T06:33:21.924Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /var/backups
2025-11-08T06:33:21.944Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /var/lib/docker
2025-11-08T06:33:21.954Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/f0a22cf910514ea685667dcfeae4af49
2025-11-08T06:33:21.962Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/htzsb
2025-11-08T06:33:21.989Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/s3-comics
2025-11-08T06:33:21.998Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/htzsb/cubby
2025-11-08T06:33:22.008Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/3c456b65-acc6-4adb-800f-815808cd040d/data
2025-11-08T06:33:22.015Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/4f598cd8-bc25-4d3b-81be-dc2a8940ca88/data
2025-11-08T06:33:22.022Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/5a33ded6-18cb-48ef-8f12-8f005e85a4ac/data
2025-11-08T06:33:22.028Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/5a973be7-bf99-4a6b-a86e-0c9255dbedea/data
2025-11-08T06:33:22.033Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/66657705-2986-47e6-bc39-ab0fe97ffd88/data
2025-11-08T06:33:22.040Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/699c5175-8622-42e0-9160-ca492b447ff3/data
2025-11-08T06:33:22.044Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/6a30aea2-bd41-45e4-bce8-0857cc820e98/data
2025-11-08T06:33:22.049Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/6e2a1d58-9d9f-4b11-ba7b-54350be42f07/data
2025-11-08T06:33:22.055Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/6f00af10-595b-4534-a18e-c87af0a07fbd/data
2025-11-08T06:33:22.060Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/733cef35-842d-42a4-a370-361059dda0df/data
2025-11-08T06:33:22.066Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/73f5d39a-87eb-47a9-bcb5-0c46fe0e716b/data
2025-11-08T06:33:22.074Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/8aefd77b-c71a-4454-99e2-34e9f8c2d760/data
2025-11-08T06:33:22.081Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/8d6876a3-cf18-4c4c-b963-2ad563a88f7b/data
2025-11-08T06:33:22.089Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/htzsb/photos/immich
2025-11-08T06:33:22.107Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/b67cc23a-69b4-41e3-ac29-2303df9dc361/data
2025-11-08T06:33:22.116Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/bf2a2bb5-f382-456b-9199-658074a588c5/data
2025-11-08T06:33:22.125Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/c9f7d3ba-2e4c-41f4-9e57-cf1ecf8369ed/data
2025-11-08T06:33:22.135Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/e40218cb-9da0-44cc-8bfa-46b6003499ba/data
2025-11-08T06:33:22.143Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/f0afb254-3680-4818-a073-b7a69bdf5357/data
2025-11-08T06:33:22.151Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/f39aa622-2502-4b1e-ac88-8949ad4a3d97/data
2025-11-08T06:33:22.162Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/f582f665-f654-4dba-942d-7708aa8b0fca/data
2025-11-08T06:33:22.172Z box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name
2025-11-08T06:33:22.176Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /apps.swap
2025-11-08T06:33:23.270Z box:shell metrics: lsblk -ndo PKNAME /dev/sda1
2025-11-08T06:33:23.279Z box:shell metrics: lsblk -ndo PKNAME /dev/sda
2025-11-08T07:25:26.135Z box:logs tail: failed to kill children kill-child: target PID 2507 does not exist.
 
2025-11-08T07:42:08.379Z box:logs tail: failed to kill children kill-child: target PID 2699 does not exist.
 
2025-11-08T09:03:07.917Z box:box Received SIGTERM. Shutting down.
2025-11-08T09:03:07.918Z box:platform uninitializing platform
2025-11-08T09:03:07.920Z box:platform onDeactivated: stopping post activation services
2025-11-08T09:03:07.921Z box:tasks stopAllTasks: 0 tasks are running. sending abort signal
2025-11-08T09:03:07.921Z box:shell tasks: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/stoptask.sh all
2025-11-08T09:03:08.033Z box:database pool closed
2025-11-08T09:03:10.037Z box:box Shutdown complete
2025-11-08T09:03:38.683Z box:server ==========================================
2025-11-08T09:03:38.684Z box:server            Cloudron 9.0.7  
2025-11-08T09:03:38.684Z box:server ==========================================
2025-11-08T09:03:38.685Z box:platform initialize: start platform
2025-11-08T09:03:38.686Z box:tasks stopAllTasks: 0 tasks are running. sending abort signal
2025-11-08T09:03:38.687Z box:shell tasks: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/stoptask.sh all
2025-11-08T09:03:38.814Z box:locks releaseAll: all locks released
2025-11-08T09:03:38.823Z box:reverseproxy writeDashboardConfig: writing dashboard config for p2lab.com
2025-11-08T09:03:38.850Z box:shell reverseproxy: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/restartservice.sh nginx
2025-11-08T09:03:39.397Z box:platform onActivated: starting post activation services
2025-11-08T09:03:39.397Z box:platform startInfra: checking infrastructure
2025-11-08T09:03:39.398Z box:platform startInfra: updating infrastructure from 49.8.0 to 49.9.0
2025-11-08T09:03:39.398Z box:platform markApps: reconfiguring apps
2025-11-08T09:03:39.398Z box:reverseproxy removeAppConfigs: removing app nginx configs
2025-11-08T09:03:39.470Z box:platform removeAllContainers: removing all containers for infra upgrade
2025-11-08T09:03:39.470Z box:shell platform: docker ps -qa --filter label=isCloudronManaged
2025-11-08T09:03:39.622Z box:platform createDockerNetwork: recreating docker network
2025-11-08T09:03:39.622Z box:shell platform: docker network rm -f cloudron
2025-11-08T09:03:39.676Z box:shell platform: docker network rm -f cloudron errored BoxError: docker exited with code 1 signal null
    at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:82:23)
    at ChildProcess.emit (node:events:519:28)
    at maybeClose (node:internal/child_process:1101:16)
    at Socket.<anonymous> (node:internal/child_process:456:11)
    at Socket.emit (node:events:519:28)
    at Pipe.<anonymous> (node:net:346:12) {
  reason: 'Shell Error',
  details: {},
  stdout: <Buffer >,
  stdoutLineCount: 0,
  stderr: <Buffer 45 72 72 6f 72 20 72 65 73 70 6f 6e 73 65 20 66 72 6f 6d 20 64 61 65 6d 6f 6e 3a 20 65 72 72 6f 72 20 77 68 69 6c 65 20 72 65 6d 6f 76 69 6e 67 20 6e ... 71 more bytes>,
  stderrLineCount: 2,
  code: 1,
  signal: null,
  timedOut: false,
  terminated: false
}
2025-11-08T09:03:39.681Z box:platform startInfra: Failed to start services. retry=false (attempt 0): docker exited with code 1 signal null
2025-11-08T09:03:39.681Z box:platform BoxError: docker exited with code 1 signal null
    at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:82:23)
    at ChildProcess.emit (node:events:519:28)
    at maybeClose (node:internal/child_process:1101:16)
    at Socket.<anonymous> (node:internal/child_process:456:11)
    at Socket.emit (node:events:519:28)
    at Pipe.<anonymous> (node:net:346:12)
2025-11-08T09:03:55.149Z box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name
2025-11-08T09:03:55.162Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target
2025-11-08T09:03:55.185Z box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name
2025-11-08T09:03:55.240Z box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/platformdata