Backups aren’t being removed in Cloudron during Cleanup, maybe a bug.

[d19dotca]

It seems there’s an issue in Cloudron 7.3.6 with regards to cleanup backups.

My understanding of the cleanup task is that it takes its list of backups and compares against the backend storage. Does a sort of comparison to ensure that all backups it’s aware of still exists on the backend storage, and when it sees some missing it then deletes it from the Cloudron backup database to stay in sync. Also presumably at this time it applies the retention period too.

In my case, I’m seeing Cloudron think there’s system backups from November and December 2022 (end of month backups), but it’s not correct as the backups on the backend storage only date back to January 2023. Why is Cloudron not removing the stale backups?

I ran a test to restore a particular app from that date as well and it of course failed as I expected it to, but this means the database of backups in Cloudron is not properly staying in sync with the backend storage.

Mar 18 17:42:55 box:backupformat/tgz tarExtract: input stream error. BoxError: Backup not found: cloudron-backups/<sub-folder>/2022-12-31-120000-658/app_umami.<host>_v2.2.4.tar.gz.enc
at S3MultipartDownloadStream._handleError (/home/yellowtent/box/src/storage/s3.js:202:26)
at Response.<anonymous> (/home/yellowtent/box/src/storage/s3.js:241:36)
at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:367:18)
at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:686:14)
at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9) {
reason: 'Not found',
details: {}

This may be related (or possibly the same) to an issue I filed earlier as well: https://forum.cloudron.io/post/60918

The backend storage if it matters here is IDrive e2. Using the s3-v4-compat backup option in Cloudron to connect to it. Backups are all successful and has been for many months. There just seems to be an issue with the cleanup process.

[girish]

This came up a couple of days earlier as well when we were trying to fix @samir 's backup situation. Currently, we lose track of old backups. And since logs and event logs are also gone, it's impossible to know why the cleanup didn't happen Could even be a remote/cloud storage issue, who knows.

Have to look into how we can make this more auditable.

[nottheend]

I am also facing the issue: Using AWS S3 as backup provider.
Cloudron shows some old entries which do not exist in S3.
And when I run "Cleanup now" manually, it seems to be checking only the validity of the dates of the entries in Cloudron but probably doesn't compare with the backup storage

[matix131997]

I have connected backups with Backblaze B2. Until yesterday, I had copies on one server that held for a month and yesterday I changed to a week and they deleted with no problem.

[nottheend]

Interesting. I changed the retention policy few days ago to:
3 daily, 4 weekly, 6 monthly

[girish]

@nottheend mm, If a backup does not exist in remote (s3), then it will remove the local db entry in the backups table as well.

[nottheend]

The remote backups (S3) do not exist, but the local db entries are still.
However, they get eventually deleted once the time come.
I already wanted to recover an app to a non existing version, but I was scared to break sth^^

[girish]

@nottheend said in Backups aren’t being removed in Cloudron during Cleanup, maybe a bug.:

I already wanted to recover an app to a non existing version, but I was scared to break sth^^

You can always create a new install of app and test it out first. See import UI - https://docs.cloudron.io/backups/#import-app-backup