Backup Integrity check runs into oom

[msbt]

Two things:

I have a Cloudron (9.1.7) with two backup destinations, one MinIO and a Hetzner Storagebox. Both are set up to backup the same things every day, 2 hours apart.

1. For whatever reason the MinIO backup (which was the default and only backup for years) lists also stopped apps even though they're not included in the backup, both in size and # of apps.

Screenshot:

2. When trying to do an integrity check on the MinIO backup, it says it crashed due to oom (ran out of memory or terminated). The server has plenty of RAM and the Storagebox integrity check works without problems.

May 14 12:32:47 tasks: startTask - starting task 12627 with options {}. logs at /home/yellowtent/platformdata/logs/tasks/12627.log
May 14 12:32:47 tasks: updating task 12627 with: {"pending":false}
May 14 12:32:47 shell: tasks: /usr/bin/sudo --non-interactive -E /home/yellowtent/box/src/scripts/starttask.sh 12627 /home/yellowtent/platformdata/logs/tasks/12627.log 0 400 0
May 14 12:32:50 apphealthmonitor: app health: 25 running / 5 stopped / 0 unresponsive
May 14 12:32:52 shell: tasks: /usr/bin/sudo --non-interactive -E /home/yellowtent/box/src/scripts/starttask.sh 12627 /home/yellowtent/platformdata/logs/tasks/12627.log 0 400 0 errored with code 9 and signal null timeout false terminated false - stdout: "Service box-task-12627 failed to run
Service box-task-12627 finished with exit code 2 and status 9
" - stderr: "Running as unit: box-task-12627.service; invocation ID: 7db618c2df95465c8db4534413f4f106
Finished with result: oom-kill
Main processes terminated with: code=killed/status=KILL
Service runtime: 4.670s
CPU time consumed: 3.777s
Memory peak: 400.0M
Memory swap peak: 55.7M
"
May 14 12:32:52 tasks: startTask: 12627 done. error: {
  message: 'Task 12627 ran out of memory or terminated',
  code: 'crashed'
}
May 14 12:32:52 tasks: setCompleted - 12627: {"error":{"message":"Task 12627 ran out of memory or terminated","code":"crashed"}}
May 14 12:32:52 tasks: updating task 12627 with: {"completed":true,"error":{"message":"Task 12627 ran out of memory or terminated","code":"crashed"}}
May 14 12:32:52 backups: startIntegrityCheck: task error. Task 12627 ran out of memory or terminated

Could be that those 2 issues are related.

[nebulon]

For 1) the stopped apps don't create new backups anymore but the last created backup of those is still being referenced with the platform. This is to allow full Cloudron restore while not losing the stopped apps.

For 2) the integrity check is actually run with a memory limit of 400MB, which is the taskrunner default as the intergrity check does not pass any options at: https://git.cloudron.io/platform/box/-/blob/master/src/backups.js?ref_type=heads#L223

We have to see if it makes sense to introduce a setting for this or if we should just tie it to the memory limit set of the backupSite.

[msbt]

1. That I know, but how would I get the second backup site to include those? Eventually I'll switch to the second backup as primary and lose the secondary one when they're in sync.

2. Right, but why does it work with the Storagebox and not MinIO, does that require more resources?

[nebulon]

If the second contains the System backups it would include those of stopped apps, where a backup was made there (since otherwise the backup data simply wouldn't be there). But I agree this is not great to migrate. I guess for now you would have to start the app, make a full system backup on the scondary backup site and then stop the app again.

The code for retrieving backup data for integrity check depeneds on format and storage type. Memory consumption is hard to predict, so presumably S3 with some kind of data on that storage just gets it over the 400MB