Upgrade from 9.2.0 to 10.0.2 interrupted by apt-daily-upgrade, leaving pending database migrations

[p44]

Hi all

I encountered an issue while upgrading Cloudron from 9.2.0 to 10.0.2 on Ubuntu 24.04.

I investigated with ChatGPT, and below you can find report that I asked to write.

I hope this can help to fix the installation script.

Thanks

---

After the upgrade, the dashboard was unreachable. Running:

sudo cloudron-support --troubleshoot

reported:

Database migrations are pending.
Last migration in DB: 20260330000000-backupSites-remove-disk-provider.js
Last migration file: 20260820150000-users-add-loginLocationsJson.js

The Box service was running, but box.log contained:

Unknown column 'apps.requiresVpn' in 'field list'

The missing column should have been created by:

20260624120000-apps-add-requiresVpn.js

Disk space and inodes were fine, and MySQL, Docker and nginx were running normally.

Looking at the update task and previous boot journal, I found this sequence:

installer: stop box service for update
installer: switching the box code
installer: calling box setup script
start: Configuring nginx
Warning! D-Bus connection terminated.
Failed to wait for response: Connection reset by peer

At approximately the same time, apt-daily-upgrade.service and unattended-upgrade started. This caused a systemd re-execution, after which:

cloudron-updater.service: Main process exited, status=1/FAILURE

It appears that the new 10.0.2 code was installed, but the setup script stopped before applying the database migrations. This left the Cloudron code and database schema out of sync.

Recovery

I temporarily stopped and runtime-masked the automatic APT services to prevent another concurrent run:

sudo systemctl stop apt-daily.timer apt-daily-upgrade.timer
sudo systemctl stop apt-daily.service apt-daily-upgrade.service packagekit.service

sudo systemctl mask --runtime \
  apt-daily.service \
  apt-daily-upgrade.service \
  packagekit.service

After confirming that no active APT or DPKG operation was running, I stopped Box and ran the troubleshooting command again:

sudo systemctl stop box
sudo cloudron-support --troubleshoot

Because Box was down, the support tool automatically reran the Cloudron startup/migration script. All pending migrations were then processed successfully, including:

Processed migration 20260624120000-apps-add-requiresVpn
...
Processed migration 20260820150000-users-add-loginLocationsJson
Done

It also restarted Box automatically.

A subsequent check showed:

[OK] dashboard is reachable via loopback
[OK] No pending database migrations
[OK] box v10.0.2 is running
[OK] Dashboard is reachable via IPv4

PASS: 29
WARN: 0
FAIL: 0

The dashboard also returned HTTP 200.

Finally, I restored the APT services:

sudo systemctl unmask --runtime \
  apt-daily.service \
  apt-daily-upgrade.service \
  packagekit.service

sudo systemctl start apt-daily.timer apt-daily-upgrade.timer

Everything is now working correctly.

Could this have been caused by a race between the Cloudron setup script and apt-daily-upgrade? Would it make sense for the updater to temporarily inhibit automatic APT jobs or automatically resume the migration phase if the detached updater service is interrupted?

[james]

Hello @p44
Thanks for starring this.
Maybe we could add a safeguard to temporarily disable the apt-daily services when a Cloudron update is started.

[p44]

@james Thanks a lot !

[girish]

I added a retry in 9918c6e8cfab02a0120a9e8a990a6bb5e69152e1 . Thanks for the report .

[p44]

Thanks for your prompt response @girish.