Unsolved Backups aren’t being removed in Cloudron during Cleanup, maybe a bug.

[d19dotca]

It seems there’s an issue in Cloudron 7.3.6 with regards to cleanup backups.

My understanding of the cleanup task is that it takes its list of backups and compares against the backend storage. Does a sort of comparison to ensure that all backups it’s aware of still exists on the backend storage, and when it sees some missing it then deletes it from the Cloudron backup database to stay in sync. Also presumably at this time it applies the retention period too.

In my case, I’m seeing Cloudron think there’s system backups from November and December 2022 (end of month backups), but it’s not correct as the backups on the backend storage only date back to January 2023. Why is Cloudron not removing the stale backups?

I ran a test to restore a particular app from that date as well and it of course failed as I expected it to, but this means the database of backups in Cloudron is not properly staying in sync with the backend storage.

Mar 18 17:42:55 box:backupformat/tgz tarExtract: input stream error. BoxError: Backup not found: cloudron-backups/<sub-folder>/2022-12-31-120000-658/app_umami.<host>_v2.2.4.tar.gz.enc
at S3MultipartDownloadStream._handleError (/home/yellowtent/box/src/storage/s3.js:202:26)
at Response.<anonymous> (/home/yellowtent/box/src/storage/s3.js:241:36)
at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:367:18)
at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:686:14)
at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9) {
reason: 'Not found',
details: {}

This may be related (or possibly the same) to an issue I filed earlier as well: https://forum.cloudron.io/post/60918

The backend storage if it matters here is IDrive e2. Using the s3-v4-compat backup option in Cloudron to connect to it. Backups are all successful and has been for many months. There just seems to be an issue with the cleanup process.

[girish]

This came up a couple of days earlier as well when we were trying to fix @samir 's backup situation. Currently, we lose track of old backups. And since logs and event logs are also gone, it's impossible to know why the cleanup didn't happen Could even be a remote/cloud storage issue, who knows.

Have to look into how we can make this more auditable.