Cloudron Forum

Hello @loudlemur

@LoudLemur said:

My concern is narrower than that. In 10.0.4, fullBackup rethrows the first app error inside its loop (if (appBackupError) throw appBackupError; in backuptask.js). So when one app's backupCommand fails, every app after it in the loop, plus mail and the system data, is not backed up at all.

That makes a whole-server restore worse than an app failing honestly. Instead of one app with a flagged, incomplete backup, it gets a server where half the apps have no fresh backup, all because of an app that may be at the far end of the list.

That is a good point.

@LoudLemur said:

So the question was really this: Could fullBackup carry on through the remaining apps, then mark the whole backup as failed and name the app that failed? The run would still fail, as you say it should, but one app would not cost all the others their backup.

Possible? Sure. How? A different topic.
We would have to mark the backup as partially faulty but keep it.
Can you create a feature request regarding this behaviour and then we can discuss it there?

Hello @loudlemur

@LoudLemur said:

Sorry for the noise, twice in one thread.

No problem and I don't mind it at all.
It is always good to exchange thoughts which leads to more knowledge for everyone and this also creates better and more stable software.

Hello @loudlemur

@LoudLemur said:

Exiting 1 on a failed dump would cost the whole server its nightly backup. Could fullBackup record a per-app failure and carry on, or raise a notification? Then packages could fail honestly instead of writing a notice file.

If the backupCommand of an app fails, the backup should fail, since if the backupCommand did not return a success this means something is missing for a real full backup.
With this sentiment, a full system backup should also fail if a single app failed.
Full system backups that include all apps are also used to restore the whole Cloudron server and if an app failed to backup correctly and still simply passes as OK, the restore of that app would cause problems.
Or did I get something wrong about what you are asking?

Hello @loudlemur
I just noticed your community app for meilisearch only after I have also published meilisearch community app.
This now caused some issues in the community app store which we need to fix.
I have removed my community app again since it caused the issues.

But I noticed some things in your community app which could be improved.

Your package is much more polished: released, tested in CI, careful upgrade handling, protection against losing the master key, and thorough docs.
But it backs up the wrong kind of file for Cloudron. Snapshots aren't portable between versions. With no restoreCommand, restoring over the same app doesn't roll back the search data, and failed backups look successful.

My package is correct on backup and restore and has the multi-tenancy tooling. It's weaker on upgrade handling and master-key safety, and it hasn't been installed on a real Cloudron yet.

Maybe you could look into these findings and we can move everything into your community app.
Here is my repo https://git.cloudron.io/playground/meilisearch-app

Hello @loudlemur

@LoudLemur said:

Request 2 stands. A slow cold-model call completes via localhost inside the container, but through the reverse proxy it is cut at 60 seconds every time.

Just raising the timeout higher is not a resilient approach.
One system might have a response time of 30s and on another maybe 60s+
So no matter to what value the timeout is set, it can always reach the timeout.

This sounds more like an design issue of the application.
If something is still working it should inform the pending request about that and not simply do nothing until a timeout is reached.

Something similar we just hit with https://forum.cloudron.io/topic/15914/large-app-backup-70gb-fails-at-rotate-copy-step-with-nosuchkey-ionos-s3-read-after-write-consistency-but-cloudron-has-no-retry
Where the IONOS proxy in front of the S3 server simply did not respond, reached the timeout of 600s while it was in reality still working.
This was because IONOS did not implement CompleteMultipartUpload of S3 correctly.
While the CompleteMultipartUpload request could take several minutes to finalize and that is why S3 should periodically sends white space characters to keep the connection from timing out.
They did not do that and thus proxy timeout was reached and caused all sort of issues.
They could have increased the timeout to 1000s, 2000s or what ever value, but it would not have fixed the actual issue.

The same can be applied here.
If a slow cold-model takes time to complete, it should report it is still working instead of doing nothing causing the proxy to throw the timeout.

Hello @loudlemur

@LoudLemur said:

the app restart-loops before it is ever ready

A failing health check does not cause the app to restart, there must be something else going on.

Hello @jdaviescoates
Interesting finding.
I did not try to reproduce this yet, but this already reads like it could be hard to reproduce.

Maybe you could do a test if possible.
If you add a new storage site, that is also rsync and a hetzner storage box and create a fresh backup maybe this issue does not exist on the new backup site then.

Since you have already indentified single files that are not deleted and not uploaded it might be good to share this information with us so we can have a look on your system to see what is actually happening.
If that is possible, please write a mail to support@cloudron.io with the details and enable remote support on your Cloudron server.

Hello @milohiss
The @wordpress-managed version 3.20.3 had an issue where old installations did not set HTTPS=on correctly and 3.20.3-1 fixes this issue.
From a SemVer standpoint 3.20.3-1 is lower than 3.20.3 so adding a changelog entry for 3.20.3-1 after 3.20.3 would be considered wrong.
A long time ago we decided to use the $VERSION-X format as means for the Cloudron app store to directly offer the -X version instead of the $VERSION so the faulty version is skipped.
Please apologize this confusion.

Hello @sponch
it should work but with DNS propagation you might run into DNS cache issues.
So similar to https://docs.cloudron.io/backups#steps you could edit your hosts file to override DNS records to ensure you still have access.
With that changing from A to B DNS provider should always work and with the Sync DNS button it will create all records.

Hello @jayonrails

The only instruction was:

james said:

To fix this please run the following command from your ssh root/sudo session:

bash /home/yellowtent/box/setup/start.sh

But you also did run:

systemctl stop box
apt install mysql-server
systemctl start box

From the log it looks like everything worked out correctly.
Please make sure to carefully read provided instructions in the future since you did run commands that where not instruced to be executed.

Please also read https://forum.cloudron.io/post/129886 which explains that the binlogs might not be purged correctly and how to clean it up.
Otherwise some binlog files might remain and block some diskspace.

Hello @milohiss

@milohiss said:

Has a WordPress Managed package containing the fix for CVE-2026-87902—WordPress 7.1.2, or an equivalent documented fix—been published?

If the update is not listed in the forum and not visible for your WordPress, no the update has not yet been published.

On your 2. questions, if automatic updates for an individual app is disabled:

@milohiss said:

updates already in progress through the scheduling process

An app update that is already actively running will not be cancled.

@milohiss said:

An available update displayed as pending

The update will be displayed in the dashboard, but will not auto update.

@milohiss said:

update already queued for execution

If queued it should be executed.
Adding, I understand queued as in 10x wordpress apps have been given the command to be updated.
3x are actively updating the other 7x are queued and waiting.
A difference would be sceduled updates.
So if 10x wordpress instances have an update avilable and the next schedule is for e.g.: 18:00 o'clock.
If automatic updates for these 10x wordpress are now disabled they should not be updated at 18:00 o'clock.

@milohiss said:

An update already running

Same as above, already running updates should not be cancled

Hello @loudlemur
Not sure if helpful, but @brutalbirdie did package https://git.cloudron.io/elias/browserless-cloudron-app/ which uses playwright chrome, msedge, chromium, firefox, webkit browsers.
If I remember correctly browserless runs all browsers inside docker with --no-sandbox

Hello @jayonrails
This is not a Freescout issue.
Please run the instructed commands so the MySQL system server has the correct configuration again.
If you don't do this, it could also lead to future issues like a wrong character set and a wrong collation for databases created in the future by apps, which in turn could lead to backup restore issues.

It is highly advised to run the commands I have provided to ensure that the MySQL system server is running as intended by Cloudron.

Hello @jayonrails and @crbear
After running the command please report if this resolved the issue for you.

note: binlogs might not be purged automatically

If that is the case for you please run:

mysql -uroot -ppassword -e "RESET BINARY LOGS AND GTIDS;"

if that also does not clean up the binlog files in /var/lib/mysql/ stop the box.service and mysql.service and delete the files manually.

Example:

systemctl stop box.service mysql.service
# this will ask you for every file for confirmation
rm -rfi /var/lib/mysql/binlog.*
rm: remove regular file '/var/lib/mysql/binlog.000003'? y
rm: remove regular file '/var/lib/mysql/binlog.000004'? y
rm: remove regular file '/var/lib/mysql/binlog.index'? y
systemctl status mysql.service box.service

Hello @jayonrails and @crbear
I have merged the two topics into this one.

---

I have reproduced the issue, and it comes from upgrading Ubuntu 24 to Ubuntu 26.
A documentation update is deployed as of writing this.

The issue is that Ubuntu upgrade does uninstall the MySQL server on the system and both of you did correctly follow the guide and run:

systemctl stop box
apt install mysql-server
systemctl start box

This installed the MySQL server with the apt package default configuration and not the Cloudron configurations.
To fix this please run the following command from your ssh root/sudo session:

bash /home/yellowtent/box/setup/start.sh

This will ensure all services on the root system including MySQL are configured correctly.

---

Thank you again for bringing this to our attention.

Hello @jayonrails
I am currently trying to reproduce if the Ubuntu 24 to Ubuntu 26 upgrade resets the my.cnf.

This indeed does look like the same issue as https://forum.cloudron.io/topic/15988/mysql-binlogs-growing-unbounded-145-gb-no-expiry-set
So maybe the upgrade from Ubuntu 24 to Ubuntu 26 resets the /etc/mysql.cnf.
The guide https://docs.cloudron.io/guides/upgrade-ubuntu-26/ explictly states to:

For all packages (mime, nginx, timesyncd, logrotate, journald, etc.), select N or O to keep your currently-installed version (default).

I will try to reproduce this.

Hello @jayonrails
Your MySQL config looks empty and that is wrong.
Is also assume there is nothing in the folders listed in the config.

Since that command also failed for you please try again without the expire_logs_days:

mysql -uroot -ppassword -e "SELECT VERSION(), @@log_bin, @@binlog_expire_logs_seconds, @@max_binlog_size, @@binlog_expire_logs_auto_purge;"

Hello @jayonrails

Since 26.02.2021 all Cloudron installation ship the MySQL system server with:

skip-log-bin

in the /etc/mysql/my.cnf file.

Is your Cloudron installation older than 26.02.2021?

Please check your /etc/mysql/my.cnf file if skip-log-bin is set or if for example:

expire_logs_days = 10
max_binlog_size = 100M

or

binlog_expire_logs_seconds = 864000
max_binlog_size = 100M

You can also run the following command to get the live configuration:

mysql -uroot -ppassword -e "SELECT VERSION(), @@log_bin, @@binlog_expire_logs_seconds, @@expire_logs_days, @@max_binlog_size, @@binlog_expire_logs_auto_purge;"

Hello @jayonrails

I will have to check some things first, please give me 10-15 minutes.