Full backup: carry on past a failing app, then keep the backup marked as partial

[LoudLemur]

Following on from @james's suggestion in https://forum.cloudron.io/post/129999 .

What happens today

In fullBackup() (src/backuptask.js, the same in 10.0.4, 10.0.5 and master), each app is backed up in turn inside a loop, and the first failure ends the whole run:

const [appBackupError, appBackupResult] = await safe(backupAppWithTag(app, ...));
...
if (appBackupError) throw appBackupError;

Mail and the system data are only backed up after the loop, so when one app fails, every app after it in the list, plus mail and the system data, gets no backup from that run. If the failure persists (an app whose backupCommand keeps failing, for example), the same thing happens every night.

Why it matters

For a whole-server restore, one app with a flagged, incomplete backup is much better than a server where half the apps, the mail and the system data have no fresh backup, all because of an app that may sit early in the list.

It also leaves app packagers with two bad choices when a backupCommand cannot produce a complete backup. Exiting non-zero is honest, but it costs every other app on the server its backup. Exiting zero protects the others, but hides the failure from the platform, so packages end up inventing their own notices (our Meilisearch package writes a BACKUP-FAILED.txt and repeats it in the log at every start). We would much rather fail honestly.

What we are asking for

1. When an app's backup fails, record the failure and carry on with the remaining apps, then mail and the system data.
2. Keep the resulting backup, but mark it as partial, and name the app or apps that failed and why.
3. Report the run as failed, with a notification naming those apps, so nothing passes as fine.
4. For the failed apps, either leave them out of the partial backup or point to their last good backup, but say which, so that a restore from a partial backup warns before it restores an app from something older or incomplete.
5. Do not let partial backups count towards retention in a way that prunes the last complete backup. Otherwise a week of partial runs could rotate out the only full one.

With that in place, packages can exit non-zero whenever their backupCommand genuinely fails, and the platform stays the one place where backup health is reported.

We are happy to test a build against a package with a deliberately failing backupCommand.

[girish]

Interesting report. Another quirk of community packages. I wrote the whole backup logic with the assumption that backup fail was something we should fix. If the did fail, either it was some platform error or some external backup service error