Cloudron Forum

@necrevistonnezr said:

What’s the VPS?

One from Greenhost. No idea why it would differ from the default then, in this regard. If pretty much every other Ubuntu installation does have wget (which I’d hope) then indeed it seems fine to ignore this (non-)issue, I might better raise it at the hoster instead.

For the record, their VPS also had another customisation that conflicted with Cloudron: by default the VPS kernel is managed outside the VPS, and /lib/modules is a tmpfs filesystem prefilled with defaults and automatically populated on each boot. Cloudron tries to install/update some kernel-related packages, which complaints about ‘no space left on device’. (I reinstalled the VPS with a ‘stock kernel’ option, which did not pose this problem.)

And another piece of feedback on installation experience: while installing, I lost my internet connection for a bit, so ssh dropped. Not sure if it’s related, but the installation appears to be half-completed and I don’t know how to continue.

The last messages on my screen (IP address redacted):

=> Checking version
=> Downloading Cloudron version 8.0.6 ...
=> Installing base dependencies (this takes some time) ...Timeout, server 123.123.123.123 not responding.

Last lines of /var/log/cloudron-setup.log:

Importing restoreConfigs
[INFO] Processed migration 20170418073650-backups-add-restoreConfigJson
[INFO] Processed migration 20170423032116-settings-set-default-retentionSecs
[INFO] Processed migration 20170628173051-settings-default-mailrelay

My login prompt does tell me:

Visit one of the following URLs and accept the self-signed certificate to finish setup.

• https://123.123.123.123

But that page does not load. (with insecure http it does give an nginx page though.)

If I rerun the setup I get little help:

Error: Cloudron is already installed. To reinstall, start afresh

Perhaps installation is indeed completed? If so I would have expected a success message at the end of the log file. (But the https server appears not to work, so I still guess something went wrong.)

PS Indeed it worked fine when reinstalling the OS, and redoing the installation steps. Though I hope some day there will be a quicker way to continue/restart the installation, and/or have it continue/pause when the shell is disconnected: Not everybody has the luxury of stable internet connectivity.

The Cloudron installation instructions say:

Create a fresh Ubuntu Noble 24.04 x64 server and run these commands

wget https://cloudron.io/cloudron-setup
chmod +x ./cloudron-setup
./cloudron-setup

However my fresh ubuntu server answers:
-bash: wget: command not found

It’s easily fixed by first running apt install wget, but I imagine that for some of cloudron’s target audience this might be a showstopper.

(Note I’m not sure if the Ubuntu my VPS provider offers is less equipped than other people’s Ubuntu..)

For statistics, had the same issue here.

2023-02-08T13:57:54.944Z box:tasks setCompleted - 6496: {"result":null,"error":{"stack":"BoxError: Failed to download https://s3.amazonaws.com/prod-cloudron-releases/versions.json: downloadUrl exited with code 22 signal null\n    at /home/yellowtent/box/src/updater.js:50:26\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)\n    at async promiseRetry (/home/yellowtent/box/src/promise-retry.js:17:20)\n    at async downloadUrl (/home/yellowtent/box/src/updater.js:42:5)\n    at async downloadAndVerifyRelease (/home/yellowtent/box/src/updater.js:110:5)\n    at async update (/home/yellowtent/box/src/updater.js:145:25)","name":"BoxError","reason":"Network Error","details":{},"message":"Failed to download https://s3.amazonaws.com/prod-cloudron-releases/versions.json: downloadUrl exited with code 22 signal null"}}

I deleted /home/yellowtent/platformdata/update/updatechecker.json, and also versions.json.sig from the same folder (otherwise there was no ‘check versions’ button at all in the UI). Then systemctl restart box as described, seemed to work. (to then discover other update issues, but probably unrelated..)

@girish said in Cloudron update failing:

Indeed, the docker install logic was not cleaned up properly when we cleaned up other package installs. fixed now - https://git.cloudron.io/cloudron/box/-/commit/09d3d258b61ba338c0fadc88c0418d8bcc77f528 . Thanks for investigating!

Perhaps that fixes a similar bug, but just for clarity, note that I was using an old version Cloudron (6.1.2), and got stuck in this other loop that looks like it has already been removed from the script between version 6.2 and 6.3.

while ! dpkg --force-confold --configure -a; do
    log "Failed to fix packages. Retry"
    sleep 1
done

That “Failed to fix packages” is the message that appeared again and again in my infinite logs:

2022-01-13T17:05:53 ==> installer: Updating from 6.1.2 to 6.2.7
2022-01-13T17:05:53 ==> installer: updating docker
2022-01-13T17:05:58 ==> installer: Waiting for all dpkg tasks to finish...
dpkg: dependency problems prevent configuration of linux-image-generic:
 linux-image-generic depends on linux-modules-extra-5.4.0-59-generic; however:
  Package linux-modules-extra-5.4.0-59-generic is not installed.

dpkg: error processing package linux-image-generic (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of linux-generic:
 linux-generic depends on linux-image-generic (= 5.4.0.59.62); however:
  Package linux-image-generic is not configured yet.

dpkg: error processing package linux-generic (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of linux-image-5.4.0-59-generic:
 linux-image-5.4.0-59-generic depends on linux-modules-5.4.0-59-generic; however:
  Package linux-modules-5.4.0-59-generic is not installed.

dpkg: error processing package linux-image-5.4.0-59-generic (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 linux-image-generic
 linux-generic
 linux-image-5.4.0-59-generic
2022-01-13T17:05:58 ==> installer: Failed to fix packages. Retry
dpkg: dependency problems prevent configuration of linux-image-generic:
…
…

@girish done! https://forum.cloudron.io/topic/6333/update-to-latest-button-option

A tweak (mostly UI) that would let me say “run all missing updates one after another”, would save a lot of waiting and clicking for those cases where a version of cloudron or some app was held back.

Described here and here.

@girish thanks for the quick reaction!

I understand you aim for the rolling-update approach, and I can also imagine that skipping some versions could lead to issues. But even if this stays this way, a tweak (mostly UI) that would let me say “run all missing updates one after another”, would save a lot of waiting and clicking for those cases where a version of cloudron or some app was held back. I see this has been argued for before, with more examples for why this may not be “a corner case”. I hope you will consider the option!

And another request from this case: could it be possible to have the updater jump forward multiple versions at once? My “update available” just shows the next version, then after updating I click ‘check for updates’ and it tells me there is yet another one, and so forth.. will take me an hour of clicking to get up to date again. I noticed the same thing seems to be the case when updating an individual app, rather cumbersome!

In any case, thanks still for the swift response and keep up the good work!

Ok.. the underlying cause seems to have been that my /lib/modules is a 100MB tmpfs filesystem, on which dpkg tried to unpack 250MB of files. I don’t know how/why this happened, but I suppose it is not Cloudron’s fault either. (I managed to solve things by running umount /lib/modules so I could run apt --fix-broken install without the disk space error; I’m somewhat surprised the machine managed to reboot after that)

Anyhow, the part that is relevant for Cloudron seems to be that an issue in ubuntu/dpkg/whatever while updating things can apparently end up in an infinite loop that blocks future updates while creating gigabytes of log files; without feedback to the user that something is going wrong, let alone what is going wrong. Perhaps it could already help to have a time-limit on the updater service, and/or making dpkg stop rather than keep retrying, and then the UI can report that updating failed?

So it looks like the updater service has been running some months. The latest updater log seems to match the date it started (updater/cloudron-updater-2021-11-02_05-00-56.log), and it is a 6 GB file that looks like it repeats this bunch of lines:

dpkg: dependency problems prevent configuration of linux-image-generic:
 linux-image-generic depends on linux-modules-extra-5.4.0-59-generic; however:
  Package linux-modules-extra-5.4.0-59-generic is not installed.

dpkg: error processing package linux-image-generic (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of linux-generic:
 linux-generic depends on linux-image-generic (= 5.4.0.59.62); however:
  Package linux-image-generic is not configured yet.

dpkg: error processing package linux-generic (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of linux-image-5.4.0-59-generic:
 linux-image-5.4.0-59-generic depends on linux-modules-5.4.0-59-generic; however:
  Package linux-modules-5.4.0-59-generic is not installed.

dpkg: error processing package linux-image-5.4.0-59-generic (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 linux-image-generic
 linux-generic
 linux-image-5.4.0-59-generic
2022-01-13T00:00:17 ==> installer: Failed to fix packages. Retry

I’ll try stop the service and try again, and see what happens.

Ah yes I did not notice the task has its own log file, I overlooked that line.

From that log (after some messages about downloading):

2022-01-13T13:48:49.786Z box:tasks 3371: {"percent":70,"message":"Installing update"}
2022-01-13T13:48:49.786Z box:shell update spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/update.sh /tmp/box-3242447597
2022-01-13T13:48:49.803Z box:shell update (stdout): Updating Cloudron with /tmp/box-3242447597
=> reset service cloudron-updater status (of previous update)

2022-01-13T13:48:49.809Z box:shell update (stdout): => Run installer.sh as cloudron-updater.

2022-01-13T13:48:49.818Z box:shell update (stdout): => starting service (ubuntu 18.04) cloudron-updater. see logs at /home/yellowtent/platformdata/logs/updater/cloudron-updater-2022-01-13_13-48-49.log

2022-01-13T13:48:49.826Z box:shell update (stdout): Failed to start transient service unit: Unit cloudron-updater.service already exists.

2022-01-13T13:48:49.827Z box:shell update (stdout): Failed to install cloudron. See log for details

2022-01-13T13:48:49.828Z box:shell update code: 1, signal: null
2022-01-13T13:48:49.829Z box:taskworker Task took 7.718 seconds
2022-01-13T13:48:49.829Z box:tasks setCompleted - 3371: {"result":null,"error":{"stack":"BoxError: update exited with code 1 signal null\n    at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:66:17)\n    at ChildProcess.emit (events.js:198:13)\n    at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)","name":"BoxError","reason":"Spawn Error","details":{},"message":"update exited with code 1 signal null","code":1,"signal":null}}
2022-01-13T13:48:49.829Z box:tasks 3371: {"percent":100,"result":null,"error":{"stack":"BoxError: update exited with code 1 signal null\n    at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:66:17)\n    at ChildProcess.emit (events.js:198:13)\n    at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)","name":"BoxError","reason":"Spawn Error","details":{},"message":"update exited with code 1 signal null","code":1,"signal":null}}

The main error being “Failed to start transient service unit: Unit cloudron-updater.service already exists.”

Before you ask, the mentioned /home/yellowtent/platformdata/logs/updater/cloudron-updater-2022-01-13_13-48-49.log file does not exist.

Not sure it matters that it mentions “(ubuntu 18.04) cloudron updater” while the server runs Ubuntu 20.04.1 LTS.

I guess the issue is with this service?:

# service cloudron-updater status
● cloudron-updater.service - /tmp/box-2930230092/scripts/installer.sh
     Loaded: loaded (/run/systemd/transient/cloudron-updater.service; transient)
  Transient: yes
     Active: active (running) since Tue 2021-11-02 05:00:56 UTC; 2 months 11 days ago
   Main PID: 30747 (installer.sh)
      Tasks: 2 (limit: 4660)
     Memory: 16.8M
     CGroup: /system.slice/cloudron-updater.service
             ├─  30747 /bin/bash /tmp/box-2930230092/scripts/installer.sh
             └─3797894 sleep 1

Updating Cloudron seems to have been failing for me, for a while already. I just tried run an update manually, with backups disabled, from version 6.1.2 to 6.2.7 (not sure why it does not try to jump further than that); it simply stops after some seconds, with this in the log:

Jan 13 14:48:41 box:locker Acquired : box_update
Jan 13 14:48:41 box:tasks startTask - starting task 3371. logs at /home/yellowtent/platformdata/logs/tasks/3371.log
Jan 13 14:48:41 box:shell startTask spawn: /usr/bin/sudo -S -E /home/yellowtent/box/src/scripts/starttask.sh 3371 /home/yellowtent/platformdata/logs/tasks/3371.log 15 400
Jan 13 14:48:41 box:shell startTask (stdout): Running as unit: box-task-3371.service
Jan 13 14:48:49 box:shell startTask (stdout): Finished with result: exit-code
processes terminated with: code=exited/status=50
runtime: 8.688s
Jan 13 14:48:49 box:shell startTask code: 50, signal: null
Jan 13 14:48:49 box:tasks startTask: 3371 completed with code 50 and signal null
Jan 13 14:48:49 box:locker Released : box_update
Jan 13 14:48:49 box:updater Update failed with error { stack:
'BoxError: update exited with code 1 signal null\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:66:17)\n at ChildProcess.emit (events.js:198:13)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)',
name: 'BoxError',
reason: 'Spawn Error',
details: {},
message: 'update exited with code 1 signal null',
code: 1,
signal: null }
Jan 13 14:48:49 box:tasks startTask: 3371 done

Any suggestions what to do, where to look? I could just reinstall from scratch if that seems the easier solution. I seem to be having other issues too, with backups not being cleaned up, and apps not automatically updating either; perhaps there is a deeper issue underneath all those quirks.

@girish Indeed, the pad had ‘freely’ permission; but that should mean freely editable only to people knowing the URL. By clicking a link in the pad, the pad’s URL was made available to the owner of the linked website.

It looks like Cloudron hides apps’ Referrer-Policy HTTP header (in this line), replacing it with the rather lax value “no-referrer-when-downgrade”, which can result in private URLs leaking to third parties.

Example case that led to this discovery: We run HedgeDoc (which adds a “same-origin” policy); and just this week we found that, without us having published our pad’s URL anywhere, somebody had edited the pad after finding the URL in their server logs.

I am not sure what the ideal solution would be. Perhaps you may want to add your referrer policy only if the application did not already specify one?

On GitHub Pages (the ‘real one’), if one forgets to set the url variable in jekyll’s configuration file, apparently it will automagically be set correctly.

Using Cloudron’s GitHub Pages app, the url seems to remain empty. Or, at least, the absolute_url filter in jekyll does nothing. (and setting the url variable in _config.yml fixed this for us)

@girish Thanks for the response!

For higher loads, I recommend fronting the site with something like cloudflare.

We might indeed try use such a CDN if it turns out to be needed (probably not cloudflare, to not centralise the web too much). Ideally people would all just get our site via peer-to-peer delivery instead.

@girish said in Performance:

@gerben BTW, is the site already on GitHub pages app?

No, currently our server runs a copy at web.redecentralize.org; from this speed test it looks like you’d have 1.66 second when visiting from San Francisco (to Amsterdam). Not as good as the 550ms I get for redecentralize.org (i.e. GitHub’s servers/CDN), but acceptable I guess. I see we could also trim some file sizes..

I also found https://redecentralize.org/interviews/ (fantastic!), this is going to keep me occupied for a while

Enjoy.

Hi, this drop-in replacement for Github Pages works great.

Looking at the source code, it looks like the pages are served directly by the Express framework in NodeJS; correct? I wonder how well this performs under load; any estimates/experiences? (we are considering hosting redecentralize.org on our own server, but are a little worried about e.g. a slashdot effect taking it down when it is needed most..)

The Express docs suggest running e.g. nginx as a cache in front of it, might that be a good idea? And if not, perhaps at least the app could use gzip compression?

(besides, using an nginx server might also be useful to e.g. obtain server logs for statistics, but that’s a separate question)