Updates for apps sorted by size

[jdaviescoates]

That does sound insanely expensive!

Just to also add, I'm doing daily back-ups of 49 apps that total 247.12 GB (plus a few other small Cloudrons too) and my backups (all tarball backups) comes to a total of 3.63 TB and I've got all that stored on a 5TB Hetzner that only costs €13.08/mo

[subven]

@jdaviescoates said in Updates for apps sorted by size:

That does sound insanely expensive!

Just to also add, I'm doing daily back-ups of 49 apps that total 247.12 GB (plus a few other small Cloudrons too) and my backups (all tarball backups) comes to a total of 3.63 TB and I've got all that stored on a 5TB Hetzner that only costs €13.08/mo

Same for me ^^ I also use the same storagebox for my Laptop, mobile and Desktop backups mostly done with Duplicati.

As my Cloudron grows I want to switch over to RSYNC because at the moment, TGZ is still faster because of many, many, many small files.

[marcusquinn]

Paid $20 for 5TB for a year with idrive.com/e2 sign-up offer.

[jdaviescoates]

@marcusquinn said in Updates for apps sorted by size:

Paid $20 for 5TB for a year with idrive.com/e2 sign-up offer.

Those are still the prices too:

That is insanely cheap for the first year, although I note their prices are slightly higher than Hetzner after that first year.

[privsec]

This is partly why I want to know what apps have what amount of space in backups.

I have 5.6 tb of data in wasabi, but I have no idea as to why its that high

It does look like my backups in cloudron do not list all the backups in Wasabi

MY thought process is that this is an issue/bug

[jdaviescoates]

@privsec I just want to make sure you've seen this post?

@marcusquinn said in Updates for apps sorted by size:

@privsec Ouch, that's way too much for personal use!

1. You must make sure that version history is OFF for ALL buckets.
2. They do have a 3 month data retention charge, so even if you do reduce storage, you'll still be paying for the highest amount used for another 3 months.
3. Make sure all your backups are Tarball, it uses a lot less disk space, and is just as fast, if not faster to upload.
4. Checkout idrive.com/e2, I've moved from Wasabi to that now, and happy with it for cost, especially with Tarball backups being that much less in disk space. Again, you don't need version history on any backups, that would just be duplicate data that Cloudron already covers with your retention settings.

If you haven't turned of version history for all your buckets this is likely the cause.

[marcusquinn]

@privsec Sounds like it is because you have Wasabi bucket versioning on, which is basically ballooning your account every time you backup, as it's making copies of ever file, every time. Been there, had the same issue, they are a bitch for having that gotchya setting by default, and they also won't let you delete your card details - ever - so only way not to pay on leaving them is cancelling your card. Even then, I have a feeling they'd get threatening.

[marcusquinn]

@privsec Not a Cloudron issue or bug - its a Wasabi business model and default settings costly annoyance. Sorry you found this out the hard way. I've tried to highlight a few times around the forum.

[privsec]

@jdaviescoates said in Updates for apps sorted by size:

@privsec I just want to make sure you've seen this post?

@marcusquinn said in Updates for apps sorted by size:

@privsec Ouch, that's way too much for personal use!

1. You must make sure that version history is OFF for ALL buckets.
2. They do have a 3 month data retention charge, so even if you do reduce storage, you'll still be paying for the highest amount used for another 3 months.
3. Make sure all your backups are Tarball, it uses a lot less disk space, and is just as fast, if not faster to upload.
4. Checkout idrive.com/e2, I've moved from Wasabi to that now, and happy with it for cost, especially with Tarball backups being that much less in disk space. Again, you don't need version history on any backups, that would just be duplicate data that Cloudron already covers with your retention settings.

If you haven't turned of version history for all your buckets this is likely the cause.

@marcusquinn said in Updates for apps sorted by size:

@privsec Sounds like it is because you have Wasabi bucket versioning on, which is basically ballooning your account every time you backup, as it's making copies of ever file, every time. Been there, had the same issue, they are a bitch for having that gotchya setting by default, and they also won't let you delete your card details - ever - so only way not to pay on leaving them is cancelling your card. Even then, I have a feeling they'd get threatening.

@marcusquinn said in Updates for apps sorted by size:

@privsec Not a Cloudron issue or bug - its a Wasabi business model and default settings costly annoyance. Sorry you found this out the hard way. I've tried to highlight a few times around the forum.

I have confirmed versioning is not turned on

This is my utilization overview

[subven]

@privsec said in Updates for apps sorted by size:

I have 5.6 tb of data in wasabi, but I have no idea as to why its that high

Just purge (empty) your Cloudron bucket at Wasabi. Utilisation for your account should be 0 now. Make a (manual) backup afterwards. If you use TGZ, your app backups are compressed and it should be easy to determine individual size. Also check back overall utilisation of storage within your Wasabi. Now make another backup and compare the groth per backup done. If you keep track on this process over lets say a week, you will find out whats going wrong.

PS: How many data does your Cloudron + apps store? You can find out at System Info.

EDIT: I guess I found the answer to your question.

@privsec said in Updates for apps sorted by size:

This is my utilization overview

Take a look at this: How does Wasabi's minimum storage duration policy work?

I did not read the page but at first glance I can safely assume what is going on Maybe move away from Wasabi and use other known good services like Hetzner Storagebox. In case it is not obvious: Wasabi lets you pay for data you did already delete (the black bar in the graph) in addition to data you store at the moment (green bar).

[privsec]

@subven

Disk Utilization

Encryption settings

[privsec]

@subven
I understand that Cloudron is deleting backups and Wasabi is charging for that, however, thats only part of the issue.

My backup size though should not be anywhere nere 6 TBs even if an exact image copy of my server is uploaded to wasabi at each backup.

Backup settings

The problem will simply follow me around until I discern why Cloudron has so many files "backed up" when according to cloudron, not many are.

[subven]

@privsec said in Updates for apps sorted by size:

My backup size though should not be anywhere nere 6 TBs

Where do you get the 6TB number from? Is it the green bar from your Wasabi account? Go into your invoices at Wasabi and take a look at the Deleted Storage vs Total Storage numbers.

Also: 4 backups per month = 4 x ~500GB + extra backups for plattform and app backups. This could be something around 3-4TB in actual backup size + even more "dead" (deleted) storage you use because of the retention policy.

[subven]

One of my main Cloudrons with ~75GB in appdata including mail + box.

Daily backups with 1 week retention.

Backup storage usage is 552GB in total. Some folders are older then they should be but they do not contain full backups as I also have stopped apps.

Total size for the "out of scope" folders is 57GB.

Lets do the math! Assumed backup size = 75GB * 7 days = 525 GB (without compression).
If I factor in compression it is 55-60GB * 7 days = 385-420 GB.

Real size 552GB so it is around 1/3 more then I would assume. This seems fine to me since additional plattform backups and app backups before updates are also included.

btw. this is from a 5TB Hetzner storage box

[marcusquinn]

@privsec Seriously, get yourself off of Wasabi, OK service, but sharks for bill shock. You could get yourself setup with IDrive E2, change to Tarball backups, and delete all your Wasabi buckets in the time it's taking to go through all this diagnostics. It's not a Cloudron issue, it's a Wasabi trickery issue, their interface is shit, their retention policy is annoying, their manners were never that helpful when I had the same issue.

[timconsidine]

@marcusquinn said in Updates for apps sorted by size:

Seriously, get yourself off of Wasabi, OK service, but sharks for bill shock

Absolutely
I stopped using Wasabi and use Scaleway

[privsec]

@marcusquinn I get it, totally. And I plan on it.
But the original issue remains.

Even If I switch services, without versioning I have nearly 6TB of data being stored.

I want to know where that's from so I can prune as needed.

[subven]

@privsec said in Updates for apps sorted by size:

I want to know where that's from so I can prune as needed.

Again...where do you get the 6TB number from? You can search through your backup files by domain like app_domain.com* to figure out what app consumes how much within your backup.

So in my case, 25GB compressed appdata ending up as 275GB in my backup. 1 backup per day and 7 days retention. Keep in mind that another another full mirror is stored in the backups "storage" folder under different naming.

Because you have one big app (the light blue one...is it Nextcloud?) with lots of data in it, this could be the reason for your backup size. Also consider using RSYNC if you have lots of huge files that don't change that often.

[jdaviescoates]

@privsec said in Updates for apps sorted by size:

Even If I switch services, without versioning I have nearly 6TB of data being stored.

I bet you wouldn't.

And as others have explained, just do tarball backups one by one on a new backup provider and then you'll know how much space each app's backup takes up in reality.

[privsec]

@subven

@privsec said in Updates for apps sorted by size:

This is partly why I want to know what apps have what amount of space in backups.

I have 5.6 tb of data in wasabi, but I have no idea as to why its that high

It does look like my backups in cloudron do not list all the backups in Wasabi

MY thought process is that this is an issue/bug

[privsec]

@privsec said in Updates for apps sorted by size:

@subven

Disk Utilization

Encryption settings

@jdaviescoates said in Updates for apps sorted by size:

@privsec said in Updates for apps sorted by size:

Even If I switch services, without versioning I have nearly 6TB of data being stored.

I bet you wouldn't.

And as others have explained, just do tarball backups one by one on a new backup provider and then you'll know how much space each app's backup takes up in reality.

I am currently using tarball for backups, am I missing something?

[jdaviescoates]

@privsec said in Updates for apps sorted by size:

I am currently using tarball for backups, am I missing something?

Yes you are not listening to everyone saying the same thing:

Wasabi does all sorts of dodgy jiggery pokery which is increasing your storage usage.

Just start experimenting with another backup provider to see what your actual storage usage should be.

[privsec]

@jdaviescoates
Ok, I guess I am just misunderstanding folks. I will post results with a different provider when possible.

[jdaviescoates]

@privsec I get that you want to understand what was/ is actually going on, but from where I'm sitting you spending time investigating that would just be you throwing more resources down the wasabi hole

[privsec]

@jdaviescoates I have switched to idrive just now and doing a backup. My jellyfin has all the media storage within the app container (discovered this as the backup for jelly was taking forever) I thought it was in a volume.

Now I will be looking at migrating this data to a volume.

[jdaviescoates]

@privsec said in Updates for apps sorted by size:

My jellyfin has all the media storage within the app container (discovered this as the backup for jelly was taking forever) I thought it was in a volume.

Sounds like you've likely found the main culprit!

[marcusquinn]

@privsec Good stuff, yes, Wasabi is storing duplicates of backups for 90 days after each backup, hence you are multiplying your storage use with them, even if you have backups being purged, they are showing that as still stored for another 90 days thereafter. Bit of their bullshit really, as they cannot recover deleted files, even a minute after.

[robi]

@marcusquinn and they charge for it?

[marcusquinn]

@robi Yeah. I can't recall any other S3 services doing this, but maybe they got the idea from somewhere, or maybe it's just their way of making their headline pricing appear more competitive and trying to protect revenue volatility. If it were me, I really don't think this could make as much difference to their bottom-lines as actual customer satisfaction and goodwill, because for all they might gain, I expect they lose as much in bitterness and loss of recommendation-level happiness.

[robi]

@marcusquinn their support should be able to tell you knowing how the back end works. Some systems are highly optimized for injest, but less for garbage collection, which can be deferred since it causes injest disruption. If they do this at 90day intervals, that's a pretty poor architectural set of choices.

If it's not usable, it should not be charged or passed on to the customer. They'd be smarter to hide it with larger customers where it matters less than smaller ones where it's very obvious.

[jayonrails]

@marcusquinn said in Updates for apps sorted by size:

Paid $20 for 5TB for a year with idrive.com/e2 sign-up offer.

I would like to try it. What settings do I have to appl to Cloudron backup settings to get it up and running?

[marcusquinn]

@jayonrails The generic "S3 API Compatible v4" is what's used for anything not in that list.

[jayonrails]

@marcusquinn thanks a lot, I will try it!

[jayonrails]

@marcusquinn works perfectly for me. Thanks for your tips!

[marcusquinn]

@jayonrails Cool. I recommend sticking to Tarball. Faster and takes up less space overall.