Updates for apps sorted by size

[privsec]

@subven
I understand that Cloudron is deleting backups and Wasabi is charging for that, however, thats only part of the issue.

My backup size though should not be anywhere nere 6 TBs even if an exact image copy of my server is uploaded to wasabi at each backup.

Backup settings

The problem will simply follow me around until I discern why Cloudron has so many files "backed up" when according to cloudron, not many are.

[subven]

@privsec said in Updates for apps sorted by size:

My backup size though should not be anywhere nere 6 TBs

Where do you get the 6TB number from? Is it the green bar from your Wasabi account? Go into your invoices at Wasabi and take a look at the Deleted Storage vs Total Storage numbers.

Also: 4 backups per month = 4 x ~500GB + extra backups for plattform and app backups. This could be something around 3-4TB in actual backup size + even more "dead" (deleted) storage you use because of the retention policy.

[subven]

One of my main Cloudrons with ~75GB in appdata including mail + box.

Daily backups with 1 week retention.

Backup storage usage is 552GB in total. Some folders are older then they should be but they do not contain full backups as I also have stopped apps.

Total size for the "out of scope" folders is 57GB.

Lets do the math! Assumed backup size = 75GB * 7 days = 525 GB (without compression).
If I factor in compression it is 55-60GB * 7 days = 385-420 GB.

Real size 552GB so it is around 1/3 more then I would assume. This seems fine to me since additional plattform backups and app backups before updates are also included.

btw. this is from a 5TB Hetzner storage box

[marcusquinn]

@privsec Seriously, get yourself off of Wasabi, OK service, but sharks for bill shock. You could get yourself setup with IDrive E2, change to Tarball backups, and delete all your Wasabi buckets in the time it's taking to go through all this diagnostics. It's not a Cloudron issue, it's a Wasabi trickery issue, their interface is shit, their retention policy is annoying, their manners were never that helpful when I had the same issue.

[timconsidine]

@marcusquinn said in Updates for apps sorted by size:

Seriously, get yourself off of Wasabi, OK service, but sharks for bill shock

Absolutely
I stopped using Wasabi and use Scaleway

[privsec]

@marcusquinn I get it, totally. And I plan on it.
But the original issue remains.

Even If I switch services, without versioning I have nearly 6TB of data being stored.

I want to know where that's from so I can prune as needed.

[subven]

@privsec said in Updates for apps sorted by size:

I want to know where that's from so I can prune as needed.

Again...where do you get the 6TB number from? You can search through your backup files by domain like app_domain.com* to figure out what app consumes how much within your backup.

So in my case, 25GB compressed appdata ending up as 275GB in my backup. 1 backup per day and 7 days retention. Keep in mind that another another full mirror is stored in the backups "storage" folder under different naming.

Because you have one big app (the light blue one...is it Nextcloud?) with lots of data in it, this could be the reason for your backup size. Also consider using RSYNC if you have lots of huge files that don't change that often.

[jdaviescoates]

@privsec said in Updates for apps sorted by size:

Even If I switch services, without versioning I have nearly 6TB of data being stored.

I bet you wouldn't.

And as others have explained, just do tarball backups one by one on a new backup provider and then you'll know how much space each app's backup takes up in reality.

[privsec]

@subven

@privsec said in Updates for apps sorted by size:

This is partly why I want to know what apps have what amount of space in backups.

I have 5.6 tb of data in wasabi, but I have no idea as to why its that high

It does look like my backups in cloudron do not list all the backups in Wasabi

MY thought process is that this is an issue/bug

[privsec]

@privsec said in Updates for apps sorted by size:

@subven

Disk Utilization

Encryption settings

@jdaviescoates said in Updates for apps sorted by size:

@privsec said in Updates for apps sorted by size:

Even If I switch services, without versioning I have nearly 6TB of data being stored.

I bet you wouldn't.

And as others have explained, just do tarball backups one by one on a new backup provider and then you'll know how much space each app's backup takes up in reality.

I am currently using tarball for backups, am I missing something?

[jdaviescoates]

@privsec said in Updates for apps sorted by size:

I am currently using tarball for backups, am I missing something?

Yes you are not listening to everyone saying the same thing:

Wasabi does all sorts of dodgy jiggery pokery which is increasing your storage usage.

Just start experimenting with another backup provider to see what your actual storage usage should be.

[privsec]

@jdaviescoates
Ok, I guess I am just misunderstanding folks. I will post results with a different provider when possible.

[jdaviescoates]

@privsec I get that you want to understand what was/ is actually going on, but from where I'm sitting you spending time investigating that would just be you throwing more resources down the wasabi hole

[privsec]

@jdaviescoates I have switched to idrive just now and doing a backup. My jellyfin has all the media storage within the app container (discovered this as the backup for jelly was taking forever) I thought it was in a volume.

Now I will be looking at migrating this data to a volume.

[jdaviescoates]

@privsec said in Updates for apps sorted by size:

My jellyfin has all the media storage within the app container (discovered this as the backup for jelly was taking forever) I thought it was in a volume.

Sounds like you've likely found the main culprit!

[marcusquinn]

@privsec Good stuff, yes, Wasabi is storing duplicates of backups for 90 days after each backup, hence you are multiplying your storage use with them, even if you have backups being purged, they are showing that as still stored for another 90 days thereafter. Bit of their bullshit really, as they cannot recover deleted files, even a minute after.

[robi]

@marcusquinn and they charge for it?

[marcusquinn]

@robi Yeah. I can't recall any other S3 services doing this, but maybe they got the idea from somewhere, or maybe it's just their way of making their headline pricing appear more competitive and trying to protect revenue volatility. If it were me, I really don't think this could make as much difference to their bottom-lines as actual customer satisfaction and goodwill, because for all they might gain, I expect they lose as much in bitterness and loss of recommendation-level happiness.

[robi]

@marcusquinn their support should be able to tell you knowing how the back end works. Some systems are highly optimized for injest, but less for garbage collection, which can be deferred since it causes injest disruption. If they do this at 90day intervals, that's a pretty poor architectural set of choices.

If it's not usable, it should not be charged or passed on to the customer. They'd be smarter to hide it with larger customers where it matters less than smaller ones where it's very obvious.

[jayonrails]

@marcusquinn said in Updates for apps sorted by size:

Paid $20 for 5TB for a year with idrive.com/e2 sign-up offer.

I would like to try it. What settings do I have to appl to Cloudron backup settings to get it up and running?