backup / snapshots retention

[jdaviescoates]

@rosano have you read https://docs.cloudron.io/backups/#retention-policy ?

eg

For installed apps and box backups, the latest backup is always retained regardless of the policy.

An App backup that was created right before an app updates is also marked as special and persisted for 3 weeks.

Might those explain some of what you're seeing?

[rosano]

i gather that there may be some things left beyond the retention policy, but it seems like everything or many/most things are kept forever. some of the dates in /snapshots go back to june/july of last year, over 6 months, and i have maybe only ever made a special manual snapshot or said 'keep this' once or twice. i expect most of these should be deleted by now, do i have the wrong idea?

[jdaviescoates]

@rosano I honestly don't know but it does sound like you've got stuff you shouldn't have.

I guess that could happen if your set-up/ config changed at some point - perhaps Cloudron lost access to the older ones for some reason, or something?

I've never used Backblaze, but lots of people on here seem to have add issues with them in the past...

This could well the post you're looking for:

@ericdrgn said in Easy way to "reset" backups...:

Change "file lifecycle" to "keep only the last version"?

https://forum.cloudron.io/post/28766

For more clues see e.g. these previous threads (there may be more but I quickly found these):
https://forum.cloudron.io/topic/4776/easy-way-to-reset-backups
https://forum.cloudron.io/topic/5283/backblaze-cleanup-backups-not-functioning

[joseph]

Did you change the backup provider in the middle? The backup code loses track of old backups in previous providers when you switch . If the service or server also goes unreachable for days, the backup logic loses track of things (i.e it removes things from the database and assumes stuff is gone remotely).

For the moment, you can just manually delete all the old files and see if it grown again. There were some fixed around the time of 8.1 , so it might be bugs from older cloudron version as well.

[rosano]

@jdaviescoates i didn't know about the lifecycle settings, tried it and it reduced total size by 90% after a few days

@joseph i think this was around the time i switched from aws to backblaze, but i understand it didn't transfer pre-existing backups, only starting fresh, although i could be wrong. will try deleting everything if needed, but so far the previous solution works for me

[jdaviescoates]

@rosano said in backup / snapshots retention:

@jdaviescoates i didn't know about the lifecycle settings, tried it and it reduced total size by 90% after a few days

Great to hear that sorted you out! I also just noticed that info is also already in the Cloudron docs https://docs.cloudron.io/backups/#backblaze-b2

[girish]

Was just updating the aws-sdk module to v3 in our code base last few days. I noticed that we don't clean up aborted multipart uploads. Our backup storage provider actually accumulated over 100GB of aborted uploads. For the moment, you have to set a lifecycle rule - https://aws.amazon.com/blogs/aws-cloud-financial-management/discovering-and-deleting-incomplete-multipart-uploads-to-lower-amazon-s3-costs/ .

AWS has a UI to set lifecycle rules like this but other providers like vultr/hetzner etc do not. You have to use the AWS CLI tool - aws s3api put-bucket-lifecycle-configuration --bucket my-domain-com-backups --lifecycle-configuration file://expiry.json . Next release, we will clean up aborted uploads automatically since most people forget to add this lifecycle rule / it's too complicated to know.

[JLX89]

This topic actually reminded me to take another look, and aside from what @girish mentioned about multipart uploads, I've noticed that currently I'm sitting at around 2.93 TB worth of backups, going all the way back to March 2023, when I setup this provider.

At this point, if this is the case -- is there an efficient way to create a new bucket, to start "fresh" and ensure that the retention is working correctly? Obviously, the concern is the archived apps, which I could restore first and then archive them again.

[girish]

@JLX89 instead, I recommend just setting a lifecycle rule to clean up multipart backups. It will clean it all up in a day or two based on your rule . Many providers have a UI for lifecycle rules, which one are you using?

I already added a change earlier today to clean up aborted uploads automatically - https://git.cloudron.io/platform/box/-/commit/cb6d531300870f1929dd27816d2ead316351a3c4

[JLX89]

@girish I am currently using iDrive E2. Yes, they have lifecycle policy editor. I'll have to take a look and see what baseline or default settings to setup. Thank you!

[shrey]

Hi @joseph @girish
I'm in a similar situation as well, with Cloudron v8.2.4, backups stored in DigitalOcean Spaces, via rsync.

I changed the Retention Policy to 1 month, instead of 3 months, and the older backups never got deleted.
I deleted most of them manually, but some are still left.
The Cleanup Backups process doesn't seem able to find the older/stale backups and delete.

P.S. Is this issue (https://forum.cloudron.io/post/71556) still unresolved?

[girish]

@shrey there's some cases where the backup code forgets about older backups. For example, if you switch backup providers (even temporarily). We are trying to improve the whole situation in 9.0 . For a start, with multiple providers, we can track/cleanup more accurately.

BTW, if you paste the filename of the backup in your Eventlog, does it find it there? I will look into making this more clearer i.e what is getting deleted and why not getting deleted.