Update to v6.3.4 (Ubuntu 20.04.2 LTS) fails

[M-arcus]

Hello,

the update to v6.3.5 fails when executed in the "Settings" view and the Automatic Update at 3am (it gets stuck at installing then).

The log output is:

Aug 10 10:42:27 box:locker Acquired : box_update
Aug 10 10:42:27 box:tasks startTask - starting task 6847 with options {"timeout":72000000,"nice":15,"memoryLimit":1024}. logs at /home/yellowtent/platformdata/logs/tasks/6847.log
Aug 10 10:42:27 box:shell startTask spawn: /usr/bin/sudo -S -E /home/yellowtent/box/src/scripts/starttask.sh 6847 /home/yellowtent/platformdata/logs/tasks/6847.log 15 1024
Aug 10 10:42:27 box:shell startTask (stdout): Running as unit: box-task-6847.service
Aug 10 10:42:37 box:shell startTask (stdout): Finished with result: exit-code
processes terminated with: code=exited/status=50
runtime: 10.064s
Aug 10 10:42:37 box:shell startTask code: 50, signal: null
Aug 10 10:42:37 box:tasks startTask: 6847 completed with code 50 and signal null
Aug 10 10:42:37 box:locker Released : box_update
Aug 10 10:42:37 box:updater Update failed with error {
stack: 'BoxError: update exited with code 1 signal null\n' +
' at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:74:17)\n' +
' at ChildProcess.emit (events.js:315:20)\n' +
' at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)',
name: 'BoxError',
reason: 'Spawn Error',
details: {},
message: 'update exited with code 1 signal null',
code: 1,
signal: null
Aug 10 10:42:37 box:tasks startTask: 6847 done

journalctl -f output:

Aug 10 08:42:37 my.domain.com systemd[1]: box-task-6847.service: Main process exited, code=exited, status=50/n/a
Aug 10 08:42:37 my.domain.com systemd[1]: box-task-6847.service: Failed with result 'exit-code'.

I tried to reboot, update all other boxes, execute apt -f install, dpkg --configure -a, restart the box by systemctl restart box and /home/yellowtent/box/setup/start.sh, sadly nothing worked.

When executing apt -f install there was a lock problem though, but deleting the lock didn't help either:

 apt -f install
Waiting for cache lock: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 216171 (apt)      
Reading package lists... Done                                                                             
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  btrfs-progs libargon2-0 linux-image-4.15.0-147-generic linux-modules-4.15.0-147-generic linux-modules-extra-4.15.0-147-generic
Use 'apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 29 not upgraded.

What are other possible solutions?

[girish]

@m-arcus Does /home/yellowtent/platformdata/logs/tasks/6847.log have any crash/backtrace?

[M-arcus]

@girish

2021-08-10T08:42:37.896Z box:shell update (stdout): => Run installer.sh as cloudron-updater.

2021-08-10T08:42:37.917Z box:shell update (stdout): => starting service (ubuntu 18.04) cloudron-updater. see logs at /home/yellowtent/platformdata/logs/updater/cloudron-updater-2021-08-10_08-42-37.log

2021-08-10T08:42:37.934Z box:shell update (stdout): Failed to start transient service unit: Unit cloudron-updater.service already exists.

2021-08-10T08:42:37.936Z box:shell update (stdout): Failed to install cloudron. See log for details

[girish]

@m-arcus another log file to follow. Can you see /home/yellowtent/platformdata/logs/updater/cloudron-updater-2021-08-10_08-42-37.log ?

[M-arcus]

@girish said in Update to v6.3.4 (Ubuntu 20.04.2 LTS) fails:

/home/yellowtent/platformdata/logs/updater/cloudron-updater-2021-08-10_08-42-37.log

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  btrfs-progs docker-scan-plugin libargon2-0 linux-image-4.15.0-147-generic
  linux-modules-4.15.0-147-generic linux-modules-extra-4.15.0-147-generic
Use 'apt autoremove' to remove them.
Suggested packages:
  aufs-tools cgroupfs-mount | cgroup-lite
Recommended packages:
  docker-ce-rootless-extras
The following packages will be DOWNGRADED:
  containerd.io docker-ce docker-ce-cli
0 upgraded, 0 newly installed, 3 downgraded, 0 to remove and 37 not upgraded.
E: Packages were downgraded and -y was used without --allow-downgrades.
2021-08-11T08:35:17 ==> installer: Failed to install docker. Retry

[nebulon]

@m-arcus ah loooks like somehow the system and thus docker has triggered an update of the ubuntu system outside of Cloudron. Since only security updates are enabled, maybe you have accidentally updated docker via SSH manually.

Essentially you have to downgrade docker manually now using apt prior to the update.

[M-arcus]

@nebulon @girish
To what version?

I updated the server to 20.04 from 16.04 a month ago

[M-arcus]

It was 20.10.3, i had 20.10.7 installed.

I executed:

curl -sL "https://download.docker.com/linux/ubuntu/dists/focal/pool/stable/amd64/containerd.io_1.4.3-1_amd64.deb" -o /tmp/containerd.deb
curl -sL "https://download.docker.com/linux/ubuntu/dists/focal/pool/stable/amd64/docker-ce-cli_20.10.3~3-0~ubuntu-focal_amd64.deb" -o /tmp/docker-ce-cli.deb
    curl -sL "https://download.docker.com/linux/ubuntu/dists/focal/pool/stable/amd64/docker-ce_20.10.3~3-0~ubuntu-focal_amd64.deb" -o /tmp/docker.deb
apt install -y --allow-downgrades /tmp/containerd.deb  /tmp/docker-ce-cli.deb /tmp/docker.deb

and then rebooted. Now it's installing the update to 6.3.5.

Maybe there should be --allow-downgrades option in the cloudron box installer script

[girish]

@m-arcus Did you install docker manually? I don't see how docker can get upgraded because we install docker debian packages and not from an apt repo.

[M-arcus]

@girish @nebulon
I did not follow the upgrade tutorial for 16.04 → 18.04 → 20.04 entirely because I have some minor adjustments on the server that were blocking the upgrade. Also, there was a minor issue with docker in the past, where I edited the file /etc/apt/sources.list.d/docker.list so this whole problem was my fault