Solved Disk almost full since external backup (Ubuntu / Swap)

[meaculpa]

Hey,

I made my first external backup (6+ apps) to Backblaze using encrypted tarballs for my apps, and my local server backup space was freed, but now my "Everything else (Ubuntu, Swap...)" disk usage grew +200 gb... I guess these are the local compressed files?

Will this space be freed on the next backup cycle? or will it freeze my disk? im a bit concerned and switched to rsync for now but it might result in higher backup costs.

Do I have to free this space manually? If yes, where can I find the potential disk fillers?

I dont want to recover from full disk

Thanks for the help!!

[nebulon]

The backup tarballs are streamed so they are not stashed on the disk as such. Once you have made a full backup to Backblaze, you have to cleanup the old backups on the filesystem manually. This is not done automatically since deleting backups is always a bit dangerous without the user being aware of. If you have not specially configured the backup location in the past, then those would be in /var/backups. Again only after a full backup to Backblaze was made, it is safe to delete all files in that folder.

[meaculpa]

Thank you very much for clarifying!

[meaculpa]

Hey @nebulon ,
the local storage problem was solved, but now the backups fill up on Backblaze: old backups are not getting deleted from my bucket automatically. I have my retention policy set to 2 days, but older backups remain on the cloud storage. Did I miss something again, or is this automatic behaviour not intended?
Thanks for your help again!

[girish]

@meaculpa If you go to Backups -> Clean Backups and then check the logs (the button in the section below it), do you see any errors?

[meaculpa]

@girish I just waited another backup cycle to doubl check, but the backups really pile up on Backblaze.

The Cloudron Backup Listing shows only the recent two Backups (I have 1 backup per day and 2 days retention policy configured).

When i clean manually, like you suggested it does not throw any errors, I just get

box:tasks setCompleted - 277: {"result":{"removedBoxBackupIds":[],"removedAppBackupIds":[]},"error":null}

To throw in some numbers, I have ~110gb occupied on my server, and the bucket on Backblaze is at ~600gb.

I use encrypted tarballs.

If I can do anything ot help debug, pleas let me know!

[girish]

@meaculpa said in Disk almost full since external backup (Ubuntu / Swap):

The Cloudron Backup Listing shows only the recent two Backups (I have 1 backup per day and 2 days retention policy configured).

Oh, that sounds like maybe the backblaze backend is not deleting stuff. Let me quickly test if there is some obvious bug before we can dig into some of your logs further.

[girish]

@meaculpa The delete seems to work fine from Cloudron side but it seems that backblaze has quite a few quirks!

• First, if you delete, it doesn't delete immediately. The file tree which you see in the backblaze UI may not be the latest files even. They might be gone. I can confirm this because if you try to delete something from their dashboard, it shows this warning:

• Second, each bucket also has a life cycle rule which defaults to keep all versions. Reading the lifecycle docs, it says "This is very helpful, because it means the old versions are there if the original file is accidentally deleted.". So, in the lifecycle UI below, you have to select the third one and select 1 day. That way I think the deleted files will be truly removed after a day.

In my b2 ui, i see this:

I haven't seen an explanation anywhere but I think '*' means deleted but it will be kept around based on lifecycle settings.

[meaculpa]

@girish Holy moly, this is why they charge less than other providers, cause they keep all the data around forever as the default.

Thank you very much for helping me solve this third-party issue! Maybe this hint could be added to the backup docs.

I made the settings according to your suggestion and will observe if it works.

Thanks again!!

[meaculpa]

I can confirm that, now that the lifecycle settings are correct, backups get physically deleted from B2.