backup / snapshots retention

[rosano]

my backblaze storage bill seems to be increasing every month and i think it's due to cloudron backups. my retention policy is 1 month but it seems like older backups / snapshots are not getting deleted:

i'm confused also looking at this directory as the main folder seems time-based and the subfolder /snapshot seems app-based. this subfolder seems to also have many old versions beyond retention policy.

manually cleaning this up is very cumbersome in backblaze as their checkboxes are not html but pictures or something, so one must literally click all of them. i tried cloudron's 'cleanup backups' which seems to only consider ones within my 1 month retention policy. not sure if it's aware of the files beyond that.

Feb 07 08:01:17 box:backupcleaner applyBackupRetention: 2025-01-21-040001-622/app_XXXXX_v2.26.0 keep/discard: referenced
Feb 07 08:01:17 box:backupcleaner applyBackupRetention: 2025-01-20-100002-716/app_XXXXX_v2.25.0 keep/discard: preserveSecs
Feb 07 08:01:17 box:backupcleaner applyBackupRetention: 2025-01-20-040001-314/app_XXXXX_v2.25.0 keep/discard: referenced
Feb 07 08:01:17 box:backupcleaner applyBackupRetention: 2025-01-19-040001-285/app_XXXXX_v2.25.0 keep/discard: referenced
Feb 07 08:01:17 box:backupcleaner applyBackupRetention: 2025-01-18-040001-307/app_XXXXX_v2.25.0 keep/discard: referenced
Feb 07 08:01:17 box:backupcleaner applyBackupRetention: 2025-01-17-040001-202/app_XXXXX_v2.25.0 keep/discard: referenced
Feb 07 08:01:17 box:backupcleaner applyBackupRetention: 2025-01-16-040001-480/app_XXXXX_v2.25.0 keep/discard: referenced
Feb 07 08:01:17 box:backupcleaner applyBackupRetention: 2025-01-15-040001-449/app_XXXXX_v2.25.0 keep/discard: referenced
Feb 07 08:01:17 box:backupcleaner applyBackupRetention: 2025-01-14-040001-497/app_XXXXX_v2.25.0 keep/discard: referenced
Feb 07 08:01:17 box:backupcleaner applyBackupRetention: 2025-01-13-100004-538/app_XXXXX_v2.24.0 keep/discard: keepWithinSecs
Feb 07 08:01:17 box:backupcleaner applyBackupRetention: 2025-01-13-040001-279/app_XXXXX_v2.24.0 keep/discard: referenced
Feb 07 08:01:17 box:backupcleaner applyBackupRetention: 2025-01-12-040001-348/app_XXXXX_v2.24.0 keep/discard: referenced
Feb 07 08:01:17 box:backupcleaner applyBackupRetention: 2025-01-11-040001-615/app_XXXXX_v2.24.0 keep/discard: referenced
Feb 07 08:01:17 box:backupcleaner applyBackupRetention: 2025-01-10-040001-315/app_XXXXX_v2.24.0 keep/discard: referenced
Feb 07 08:01:17 box:backupcleaner applyBackupRetention: 2025-01-09-040001-222/app_XXXXX_v2.24.0 keep/discard: referenced
Feb 07 08:01:17 box:backupcleaner applyBackupRetention: 2025-01-08-040001-314/app_XXXXX_v2.24.0 keep/discard: referenced
Feb 07 08:01:17 box:backupcleaner cleanupAppBackups: applying retention for appId 1bba7e1f-a2f5-4bd9-b2be-3a60d0153669 retention: {"keepLatest":true,"keepWithinSecs":2592000}

i didn't find any solutions in these related topics:

• Backups before retention policy not being deleted. Bug?
• Scaleway backups retained > retention policy
• Backup retention time

any suggestions?

[jdaviescoates]

@rosano have you read https://docs.cloudron.io/backups/#retention-policy ?

eg

For installed apps and box backups, the latest backup is always retained regardless of the policy.

An App backup that was created right before an app updates is also marked as special and persisted for 3 weeks.

Might those explain some of what you're seeing?

[rosano]

i gather that there may be some things left beyond the retention policy, but it seems like everything or many/most things are kept forever. some of the dates in /snapshots go back to june/july of last year, over 6 months, and i have maybe only ever made a special manual snapshot or said 'keep this' once or twice. i expect most of these should be deleted by now, do i have the wrong idea?

[jdaviescoates]

@rosano I honestly don't know but it does sound like you've got stuff you shouldn't have.

I guess that could happen if your set-up/ config changed at some point - perhaps Cloudron lost access to the older ones for some reason, or something?

I've never used Backblaze, but lots of people on here seem to have add issues with them in the past...

This could well the post you're looking for:

@ericdrgn said in Easy way to "reset" backups...:

Change "file lifecycle" to "keep only the last version"?

https://forum.cloudron.io/post/28766

For more clues see e.g. these previous threads (there may be more but I quickly found these):
https://forum.cloudron.io/topic/4776/easy-way-to-reset-backups
https://forum.cloudron.io/topic/5283/backblaze-cleanup-backups-not-functioning

[joseph]

Did you change the backup provider in the middle? The backup code loses track of old backups in previous providers when you switch . If the service or server also goes unreachable for days, the backup logic loses track of things (i.e it removes things from the database and assumes stuff is gone remotely).

For the moment, you can just manually delete all the old files and see if it grown again. There were some fixed around the time of 8.1 , so it might be bugs from older cloudron version as well.

[rosano]

@jdaviescoates i didn't know about the lifecycle settings, tried it and it reduced total size by 90% after a few days

@joseph i think this was around the time i switched from aws to backblaze, but i understand it didn't transfer pre-existing backups, only starting fresh, although i could be wrong. will try deleting everything if needed, but so far the previous solution works for me

[jdaviescoates]

@rosano said in backup / snapshots retention:

@jdaviescoates i didn't know about the lifecycle settings, tried it and it reduced total size by 90% after a few days

Great to hear that sorted you out! I also just noticed that info is also already in the Cloudron docs https://docs.cloudron.io/backups/#backblaze-b2

[girish]

Was just updating the aws-sdk module to v3 in our code base last few days. I noticed that we don't clean up aborted multipart uploads. Our backup storage provider actually accumulated over 100GB of aborted uploads. For the moment, you have to set a lifecycle rule - https://aws.amazon.com/blogs/aws-cloud-financial-management/discovering-and-deleting-incomplete-multipart-uploads-to-lower-amazon-s3-costs/ .

AWS has a UI to set lifecycle rules like this but other providers like vultr/hetzner etc do not. You have to use the AWS CLI tool - aws s3api put-bucket-lifecycle-configuration --bucket my-domain-com-backups --lifecycle-configuration file://expiry.json . Next release, we will clean up aborted uploads automatically since most people forget to add this lifecycle rule / it's too complicated to know.

[JLX89]

This topic actually reminded me to take another look, and aside from what @girish mentioned about multipart uploads, I've noticed that currently I'm sitting at around 2.93 TB worth of backups, going all the way back to March 2023, when I setup this provider.

At this point, if this is the case -- is there an efficient way to create a new bucket, to start "fresh" and ensure that the retention is working correctly? Obviously, the concern is the archived apps, which I could restore first and then archive them again.

[girish]

@JLX89 instead, I recommend just setting a lifecycle rule to clean up multipart backups. It will clean it all up in a day or two based on your rule . Many providers have a UI for lifecycle rules, which one are you using?

I already added a change earlier today to clean up aborted uploads automatically - https://git.cloudron.io/platform/box/-/commit/cb6d531300870f1929dd27816d2ead316351a3c4

[JLX89]

@girish I am currently using iDrive E2. Yes, they have lifecycle policy editor. I'll have to take a look and see what baseline or default settings to setup. Thank you!

[shrey]

Hi @joseph @girish
I'm in a similar situation as well, with Cloudron v8.2.4, backups stored in DigitalOcean Spaces, via rsync.

I changed the Retention Policy to 1 month, instead of 3 months, and the older backups never got deleted.
I deleted most of them manually, but some are still left.
The Cleanup Backups process doesn't seem able to find the older/stale backups and delete.

P.S. Is this issue (https://forum.cloudron.io/post/71556) still unresolved?

[girish]

@shrey there's some cases where the backup code forgets about older backups. For example, if you switch backup providers (even temporarily). We are trying to improve the whole situation in 9.0 . For a start, with multiple providers, we can track/cleanup more accurately.

BTW, if you paste the filename of the backup in your Eventlog, does it find it there? I will look into making this more clearer i.e what is getting deleted and why not getting deleted.