cloudron-updater-2023-09-30_02-16-06.log

will write you an email now then. thank you

• rm /home/yellowtent/platformdata/update/*
• systemctl restart box
• Refresh the dashboard and check for updates and update

Does that work?

02.07.2023	cron	Cloudron update errored. Error: Version info mismatch
02.07.2023	cron	Backup cleaner removed 1 backups
02.07.2023	cron	Cloudron update to version 7.4.3 was started
02.07.2023	cron	Cloudron updated to version 7.4.3
02.07.2023	boot	Cloudron started with version 7.4.3
01.07.2023	cron	Cloudron update to version 7.4.3 was started

this is the version i am stuck in, i dont know if this helps

Cloudron update errored. Error: update exited with code 1 signal null

{
  "taskId": "12181",
  "errorMessage": "update exited with code 1 signal null",
  "timedOut": false
}

2023-10-27T06:51:40.840Z box:settings initCache: pre-load settings
2023-10-27T06:51:40.875Z box:taskworker Starting task 12181. Logs are at /home/yellowtent/platformdata/logs/tasks/12181.log
2023-10-27T06:51:40.878Z box:tasks update 12181: {"percent":1,"message":"Checking disk space"}
2023-10-27T06:51:40.888Z box:tasks update 12181: {"percent":5,"message":"Downloading and verifying release"}
2023-10-27T06:51:40.890Z box:updater Downloading https://releases.cloudron.io/versions.json to /home/yellowtent/platformdata/update/versions.json
2023-10-27T06:51:40.890Z box:updater downloadUrl: curl -s --fail https://releases.cloudron.io/versions.json -o /home/yellowtent/platformdata/update/versions.json
2023-10-27T06:51:40.891Z box:shell downloadUrl spawn: /usr/bin/curl -s --fail https://releases.cloudron.io/versions.json -o /home/yellowtent/platformdata/update/versions.json
2023-10-27T06:51:41.723Z box:updater downloadUrl: downloaded https://releases.cloudron.io/versions.json to /home/yellowtent/platformdata/update/versions.json
2023-10-27T06:51:41.724Z box:updater Downloading https://releases.cloudron.io/versions.json.sig to /home/yellowtent/platformdata/update/versions.json.sig
2023-10-27T06:51:41.724Z box:updater downloadUrl: curl -s --fail https://releases.cloudron.io/versions.json.sig -o /home/yellowtent/platformdata/update/versions.json.sig
2023-10-27T06:51:41.724Z box:shell downloadUrl spawn: /usr/bin/curl -s --fail https://releases.cloudron.io/versions.json.sig -o /home/yellowtent/platformdata/update/versions.json$
2023-10-27T06:51:42.130Z box:updater downloadUrl: downloaded https://releases.cloudron.io/versions.json.sig to /home/yellowtent/platformdata/update/versions.json.sig
2023-10-27T06:51:42.130Z box:updater gpgVerify: /usr/bin/gpg --status-fd 1 --no-default-keyring --keyring /home/yellowtent/box/src/releases.gpg --verify /home/yellowtent/platform$
2023-10-27T06:51:42.131Z box:shell gpgVerify exec: /usr/bin/gpg --status-fd 1 --no-default-keyring --keyring /home/yellowtent/box/src/releases.gpg --verify /home/yellowtent/platf$
2023-10-27T06:51:42.162Z box:shell gpgVerify (stdout): [GNUPG:] NEWSIG
[GNUPG:] KEY_CONSIDERED 0EADB19CDDA23CD0FE71E3470A372F8703C493CC 0
[GNUPG:] SIG_ID 16eqPpSP7Z+dzlFeM3kBN5aHU8k 2023-09-28 1695916582
[GNUPG:] KEY_CONSIDERED 0EADB19CDDA23CD0FE71E3470A372F8703C493CC 0
[GNUPG:] GOODSIG 0A372F8703C493CC Cloudron UG <admin@cloudron.io>
[GNUPG:] VALIDSIG 0EADB19CDDA23CD0FE71E3470A372F8703C493CC 2023-09-28 1695916582 0 4 0 1 10 00 0EADB19CDDA23CD0FE71E3470A372F8703C493CC
[GNUPG:] TRUST_UNDEFINED 0 pgp
[GNUPG:] VERIFICATION_COMPLIANCE_MODE 23

2023-10-27T06:51:42.162Z box:shell gpgVerify (stderr): gpg: Signature made Thu 28 Sep 2023 03:56:22 PM UTC
gpg:                using RSA key 0EADB19CDDA23CD0FE71E3470A372F8703C493CC
gpg: Good signature from "Cloudron UG <admin@cloudron.io>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 0EAD B19C DDA2 3CD0 FE71  E347 0A37 2F87 03C4 93CC

2023-10-27T06:51:42.165Z box:updater Downloading https://releases.cloudron.io/box-7fe2de448e-7fe2de448e-7.5.2.tar.gz to /home/yellowtent/platformdata/update/box.tar.gz
2023-10-27T06:51:42.165Z box:updater downloadUrl: curl -s --fail https://releases.cloudron.io/box-7fe2de448e-7fe2de448e-7.5.2.tar.gz -o /home/yellowtent/platformdata/update/box.t$
2023-10-27T06:51:42.165Z box:shell downloadUrl spawn: /usr/bin/curl -s --fail https://releases.cloudron.io/box-7fe2de448e-7fe2de448e-7.5.2.tar.gz -o /home/yellowtent/platformdata$
2023-10-27T06:51:44.135Z box:updater downloadUrl: downloaded https://releases.cloudron.io/box-7fe2de448e-7fe2de448e-7.5.2.tar.gz to /home/yellowtent/platformdata/update/box.tar.gz
2023-10-27T06:51:44.136Z box:updater Downloading https://releases.cloudron.io/box-7fe2de448e-7fe2de448e-7.5.2.tar.gz.sig to /home/yellowtent/platformdata/update/box.tar.gz.sig
2023-10-27T06:51:44.136Z box:updater downloadUrl: curl -s --fail https://releases.cloudron.io/box-7fe2de448e-7fe2de448e-7.5.2.tar.gz.sig -o /home/yellowtent/platformdata/update/b$
2023-10-27T06:51:44.136Z box:shell downloadUrl spawn: /usr/bin/curl -s --fail https://releases.cloudron.io/box-7fe2de448e-7fe2de448e-7.5.2.tar.gz.sig -o /home/yellowtent/platform$
2023-10-27T06:51:44.534Z box:updater downloadUrl: downloaded https://releases.cloudron.io/box-7fe2de448e-7fe2de448e-7.5.2.tar.gz.sig to /home/yellowtent/platformdata/update/box.t$
2023-10-27T06:51:44.534Z box:updater gpgVerify: /usr/bin/gpg --status-fd 1 --no-default-keyring --keyring /home/yellowtent/box/src/releases.gpg --verify /home/yellowtent/platform$
2023-10-27T06:51:44.535Z box:shell gpgVerify exec: /usr/bin/gpg --status-fd 1 --no-default-keyring --keyring /home/yellowtent/box/src/releases.gpg --verify /home/yellowtent/platf$
2023-10-27T06:51:44.835Z box:shell gpgVerify (stdout): [GNUPG:] NEWSIG
[GNUPG:] KEY_CONSIDERED 0EADB19CDDA23CD0FE71E3470A372F8703C493CC 0
[GNUPG:] SIG_ID vwqItj31YcBTqdpnuE5ghvxnZKM 2023-08-29 1693311609
[GNUPG:] KEY_CONSIDERED 0EADB19CDDA23CD0FE71E3470A372F8703C493CC 0
[GNUPG:] GOODSIG 0A372F8703C493CC Cloudron UG <admin@cloudron.io>
[GNUPG:] VALIDSIG 0EADB19CDDA23CD0FE71E3470A372F8703C493CC 2023-08-29 1693311609 0 4 0 1 10 00 0EADB19CDDA23CD0FE71E3470A372F8703C493CC
[GNUPG:] TRUST_UNDEFINED 0 pgp
[GNUPG:] VERIFICATION_COMPLIANCE_MODE 23

2023-10-27T06:51:44.835Z box:shell gpgVerify (stderr): gpg: Signature made Tue 29 Aug 2023 12:20:09 PM UTC
gpg:                using RSA key 0EADB19CDDA23CD0FE71E3470A372F8703C493CC
gpg: Good signature from "Cloudron UG <admin@cloudron.io>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 0EAD B19C DDA2 3CD0 FE71  E347 0A37 2F87 03C4 93CC

2023-10-27T06:51:44.836Z box:updater extractTarball: tar -zxf /home/yellowtent/platformdata/update/box.tar.gz -C /tmp/box-1959666716
2023-10-27T06:51:44.836Z box:shell extractTarball spawn: /bin/tar -zxf /home/yellowtent/platformdata/update/box.tar.gz -C /tmp/box-1959666716
2023-10-27T06:51:46.541Z box:updater extractTarball: extracted /home/yellowtent/platformdata/update/box.tar.gz to /tmp/box-1959666716
2023-10-27T06:51:46.542Z box:updater Updating box with https://releases.cloudron.io/box-7fe2de448e-7fe2de448e-7.5.2.tar.gz
2023-10-27T06:51:46.542Z box:tasks update 12181: {"percent":70,"message":"Installing update"}
2023-10-27T06:51:46.542Z box:shell update spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/update.sh /tmp/box-1959666716
2023-10-27T06:51:46.595Z box:shell update (stdout): Updating Cloudron with /tmp/box-1959666716
=> reset service cloudron-updater status (of previous update)

2023-10-27T06:51:46.598Z box:shell update (stdout): => Run installer.sh as cloudron-updater.

2023-10-27T06:51:46.598Z box:shell update (stdout): => starting service cloudron-updater. see logs at /home/yellowtent/platformdata/logs/updater/cloudron-updater-2023-10-27_06-51$

2023-10-27T06:51:46.602Z box:shell update (stderr): Running as unit: cloudron-updater.service

2023-10-27T06:51:46.618Z box:shell update (stdout): cloudron-updater is still active. will check in 5 seconds

2023-10-27T06:51:51.627Z box:shell update (stdout): => cloudron-updater has failed

2023-10-27T06:51:51.630Z box:shell update code: 1, signal: null
2023-10-27T06:51:51.631Z box:taskworker Task took 10.873 seconds
2023-10-27T06:51:51.631Z box:tasks setCompleted - 12181: {"result":null,"error":{"stack":"BoxError: update exited with code 1 signal null\n    at ChildProcess.<anonymous> (/home/$
2023-10-27T06:51:51.632Z box:tasks update 12181: {"percent":100,"result":null,"error":{"stack":"BoxError: update exited with code 1 signal null\n    at ChildProcess.<anonymous> ($
BoxError: update exited with code 1 signal null
    at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:78:17)
    at ChildProcess.emit (node:events:513:28)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:293:12)

Indeed, the docker install logic was not cleaned up properly when we cleaned up other package installs. fixed now - https://git.cloudron.io/cloudron/box/-/commit/09d3d258b61ba338c0fadc88c0418d8bcc77f528 . Thanks for investigating!

Perhaps that fixes a similar bug, but just for clarity, note that I was using an old version Cloudron (6.1.2), and got stuck in this other loop that looks like it has already been removed from the script between version 6.2 and 6.3.

while ! dpkg --force-confold --configure -a; do
    log "Failed to fix packages. Retry"
    sleep 1
done

That “Failed to fix packages” is the message that appeared again and again in my infinite logs:

2022-01-13T17:05:53 ==> installer: Updating from 6.1.2 to 6.2.7
2022-01-13T17:05:53 ==> installer: updating docker
2022-01-13T17:05:58 ==> installer: Waiting for all dpkg tasks to finish...
dpkg: dependency problems prevent configuration of linux-image-generic:
 linux-image-generic depends on linux-modules-extra-5.4.0-59-generic; however:
  Package linux-modules-extra-5.4.0-59-generic is not installed.

dpkg: error processing package linux-image-generic (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of linux-generic:
 linux-generic depends on linux-image-generic (= 5.4.0.59.62); however:
  Package linux-image-generic is not configured yet.

dpkg: error processing package linux-generic (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of linux-image-5.4.0-59-generic:
 linux-image-5.4.0-59-generic depends on linux-modules-5.4.0-59-generic; however:
  Package linux-modules-5.4.0-59-generic is not installed.

dpkg: error processing package linux-image-5.4.0-59-generic (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 linux-image-generic
 linux-generic
 linux-image-5.4.0-59-generic
2022-01-13T17:05:58 ==> installer: Failed to fix packages. Retry
dpkg: dependency problems prevent configuration of linux-image-generic:
…
…

I understand you aim for the rolling-update approach, and I can also imagine that skipping some versions could lead to issues. But even if this stays this way, a tweak (mostly UI) that would let me say “run all missing updates one after another”, would save a lot of waiting and clicking for those cases where a version of cloudron or some app was held back. I see this has been argued for before, with more examples for why this may not be “a corner case”. I hope you will consider the option!

And another request from this case: could it be possible to have the updater jump forward multiple versions at once

We use a Continuous delivery/Rolling update model. We designed Cloudron to be more closer to SaaS where things are always up to date (unlike traditional enterprise selfhosted software which have LTS releases). In general, costs of supporting LTS style is too high if you make frequent releases like we do with Cloudron. But yes, as you found out, this model also does not work well for those who don't update regularly/auto-update.

In any case, thanks still for the swift response and keep up the good work!

Anyhow, the part that is relevant for Cloudron seems to be that an issue in ubuntu/dpkg/whatever while updating things can apparently end up in an infinite loop that blocks future updates while creating gigabytes of log files; without feedback to the user that something is going wrong, let alone what is going wrong. Perhaps it could already help to have a time-limit on the updater service, and/or making dpkg stop rather than keep retrying, and then the UI can report that updating failed?

dpkg: dependency problems prevent configuration of linux-image-generic:
 linux-image-generic depends on linux-modules-extra-5.4.0-59-generic; however:
  Package linux-modules-extra-5.4.0-59-generic is not installed.

dpkg: error processing package linux-image-generic (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of linux-generic:
 linux-generic depends on linux-image-generic (= 5.4.0.59.62); however:
  Package linux-image-generic is not configured yet.

dpkg: error processing package linux-generic (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of linux-image-5.4.0-59-generic:
 linux-image-5.4.0-59-generic depends on linux-modules-5.4.0-59-generic; however:
  Package linux-modules-5.4.0-59-generic is not installed.

dpkg: error processing package linux-image-5.4.0-59-generic (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 linux-image-generic
 linux-generic
 linux-image-5.4.0-59-generic
2022-01-13T00:00:17 ==> installer: Failed to fix packages. Retry

I’ll try stop the service and try again, and see what happens.

From that log (after some messages about downloading):

2022-01-13T13:48:49.786Z box:tasks 3371: {"percent":70,"message":"Installing update"}
2022-01-13T13:48:49.786Z box:shell update spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/update.sh /tmp/box-3242447597
2022-01-13T13:48:49.803Z box:shell update (stdout): Updating Cloudron with /tmp/box-3242447597
=> reset service cloudron-updater status (of previous update)

2022-01-13T13:48:49.809Z box:shell update (stdout): => Run installer.sh as cloudron-updater.

2022-01-13T13:48:49.818Z box:shell update (stdout): => starting service (ubuntu 18.04) cloudron-updater. see logs at /home/yellowtent/platformdata/logs/updater/cloudron-updater-2022-01-13_13-48-49.log

2022-01-13T13:48:49.826Z box:shell update (stdout): Failed to start transient service unit: Unit cloudron-updater.service already exists.

2022-01-13T13:48:49.827Z box:shell update (stdout): Failed to install cloudron. See log for details

2022-01-13T13:48:49.828Z box:shell update code: 1, signal: null
2022-01-13T13:48:49.829Z box:taskworker Task took 7.718 seconds
2022-01-13T13:48:49.829Z box:tasks setCompleted - 3371: {"result":null,"error":{"stack":"BoxError: update exited with code 1 signal null\n    at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:66:17)\n    at ChildProcess.emit (events.js:198:13)\n    at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)","name":"BoxError","reason":"Spawn Error","details":{},"message":"update exited with code 1 signal null","code":1,"signal":null}}
2022-01-13T13:48:49.829Z box:tasks 3371: {"percent":100,"result":null,"error":{"stack":"BoxError: update exited with code 1 signal null\n    at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:66:17)\n    at ChildProcess.emit (events.js:198:13)\n    at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)","name":"BoxError","reason":"Spawn Error","details":{},"message":"update exited with code 1 signal null","code":1,"signal":null}}

The main error being “Failed to start transient service unit: Unit cloudron-updater.service already exists.”

Before you ask, the mentioned /home/yellowtent/platformdata/logs/updater/cloudron-updater-2022-01-13_13-48-49.log file does not exist.

Not sure it matters that it mentions “(ubuntu 18.04) cloudron updater” while the server runs Ubuntu 20.04.1 LTS.

I guess the issue is with this service?:

# service cloudron-updater status
● cloudron-updater.service - /tmp/box-2930230092/scripts/installer.sh
     Loaded: loaded (/run/systemd/transient/cloudron-updater.service; transient)
  Transient: yes
     Active: active (running) since Tue 2021-11-02 05:00:56 UTC; 2 months 11 days ago
   Main PID: 30747 (installer.sh)
      Tasks: 2 (limit: 4660)
     Memory: 16.8M
     CGroup: /system.slice/cloudron-updater.service
             ├─  30747 /bin/bash /tmp/box-2930230092/scripts/installer.sh
             └─3797894 sleep 1

Also skipping backup might not be a good idea, if this is a production system.