Updates for apps sorted by size
-
That does sound insanely expensive!
Just to also add, I'm doing daily back-ups of 49 apps that total 247.12 GB (plus a few other small Cloudrons too) and my backups (all tarball backups) comes to a total of 3.63 TB and I've got all that stored on a 5TB Hetzner that only costs €13.08/mo
-
@jdaviescoates said in Updates for apps sorted by size:
That does sound insanely expensive!
Just to also add, I'm doing daily back-ups of 49 apps that total 247.12 GB (plus a few other small Cloudrons too) and my backups (all tarball backups) comes to a total of 3.63 TB and I've got all that stored on a 5TB Hetzner that only costs €13.08/mo
Same for me ^^ I also use the same storagebox for my Laptop, mobile and Desktop backups mostly done with Duplicati.
As my Cloudron grows I want to switch over to RSYNC because at the moment, TGZ is still faster because of many, many, many small files.
-
Paid $20 for 5TB for a year with idrive.com/e2 sign-up offer.
Web Design https://www.evergreen.je
Development https://brandlight.org
Life https://marcusquinn.com -
@marcusquinn said in Updates for apps sorted by size:
Paid $20 for 5TB for a year with idrive.com/e2 sign-up offer.
Those are still the prices too:
That is insanely cheap for the first year, although I note their prices are slightly higher than Hetzner after that first year.
-
This is partly why I want to know what apps have what amount of space in backups.
I have 5.6 tb of data in wasabi, but I have no idea as to why its that high
It does look like my backups in cloudron do not list all the backups in Wasabi
MY thought process is that this is an issue/bug
-
@privsec I just want to make sure you've seen this post?
@marcusquinn said in Updates for apps sorted by size:
@privsec Ouch, that's way too much for personal use!
- You must make sure that version history is OFF for ALL buckets.
- They do have a 3 month data retention charge, so even if you do reduce storage, you'll still be paying for the highest amount used for another 3 months.
- Make sure all your backups are Tarball, it uses a lot less disk space, and is just as fast, if not faster to upload.
- Checkout idrive.com/e2, I've moved from Wasabi to that now, and happy with it for cost, especially with Tarball backups being that much less in disk space. Again, you don't need version history on any backups, that would just be duplicate data that Cloudron already covers with your retention settings.
If you haven't turned of version history for all your buckets this is likely the cause.
-
@privsec Sounds like it is because you have Wasabi bucket versioning on, which is basically ballooning your account every time you backup, as it's making copies of ever file, every time. Been there, had the same issue, they are a bitch for having that gotchya setting by default, and they also won't let you delete your card details - ever - so only way not to pay on leaving them is cancelling your card. Even then, I have a feeling they'd get threatening.
-
@jdaviescoates said in Updates for apps sorted by size:
@privsec I just want to make sure you've seen this post?
@marcusquinn said in Updates for apps sorted by size:
@privsec Ouch, that's way too much for personal use!
- You must make sure that version history is OFF for ALL buckets.
- They do have a 3 month data retention charge, so even if you do reduce storage, you'll still be paying for the highest amount used for another 3 months.
- Make sure all your backups are Tarball, it uses a lot less disk space, and is just as fast, if not faster to upload.
- Checkout idrive.com/e2, I've moved from Wasabi to that now, and happy with it for cost, especially with Tarball backups being that much less in disk space. Again, you don't need version history on any backups, that would just be duplicate data that Cloudron already covers with your retention settings.
If you haven't turned of version history for all your buckets this is likely the cause.
@marcusquinn said in Updates for apps sorted by size:
@privsec Sounds like it is because you have Wasabi bucket versioning on, which is basically ballooning your account every time you backup, as it's making copies of ever file, every time. Been there, had the same issue, they are a bitch for having that gotchya setting by default, and they also won't let you delete your card details - ever - so only way not to pay on leaving them is cancelling your card. Even then, I have a feeling they'd get threatening.
@marcusquinn said in Updates for apps sorted by size:
@privsec Not a Cloudron issue or bug - its a Wasabi business model and default settings costly annoyance. Sorry you found this out the hard way. I've tried to highlight a few times around the forum.
I have confirmed versioning is not turned on
This is my utilization overview
-
@privsec said in Updates for apps sorted by size:
I have 5.6 tb of data in wasabi, but I have no idea as to why its that high
Just purge (empty) your Cloudron bucket at Wasabi. Utilisation for your account should be 0 now. Make a (manual) backup afterwards. If you use TGZ, your app backups are compressed and it should be easy to determine individual size. Also check back overall utilisation of storage within your Wasabi. Now make another backup and compare the groth per backup done. If you keep track on this process over lets say a week, you will find out whats going wrong.
PS: How many data does your Cloudron + apps store? You can find out at System Info.
EDIT: I guess I found the answer to your question.
@privsec said in Updates for apps sorted by size:
This is my utilization overview
Take a look at this: How does Wasabi's minimum storage duration policy work?
I did not read the page but at first glance I can safely assume what is going on
Maybe move away from Wasabi and use other known good services like Hetzner Storagebox. In case it is not obvious: Wasabi lets you pay for data you did already delete (the black bar in the graph) in addition to data you store at the moment (green bar). -
@subven
I understand that Cloudron is deleting backups and Wasabi is charging for that, however, thats only part of the issue.My backup size though should not be anywhere nere 6 TBs even if an exact image copy of my server is uploaded to wasabi at each backup.
Backup settings
The problem will simply follow me around until I discern why Cloudron has so many files "backed up" when according to cloudron, not many are.
-
@privsec said in Updates for apps sorted by size:
My backup size though should not be anywhere nere 6 TBs
Where do you get the 6TB number from? Is it the green bar from your Wasabi account? Go into your invoices at Wasabi and take a look at the Deleted Storage vs Total Storage numbers.
Also: 4 backups per month = 4 x ~500GB + extra backups for plattform and app backups. This could be something around 3-4TB in actual backup size + even more "dead" (deleted) storage you use because of the retention policy.
-
One of my main Cloudrons with ~75GB in appdata including mail + box.
Daily backups with 1 week retention.
Backup storage usage is 552GB in total. Some folders are older then they should be but they do not contain full backups as I also have stopped apps.
Total size for the "out of scope" folders is 57GB.
Lets do the math! Assumed backup size = 75GB * 7 days = 525 GB (without compression).
If I factor in compression it is 55-60GB * 7 days = 385-420 GB.Real size 552GB so it is around 1/3 more then I would assume. This seems fine to me since additional plattform backups and app backups before updates are also included.
btw. this is from a 5TB Hetzner storage box
-
@privsec Seriously, get yourself off of Wasabi, OK service, but sharks for bill shock. You could get yourself setup with IDrive E2, change to Tarball backups, and delete all your Wasabi buckets in the time it's taking to go through all this diagnostics. It's not a Cloudron issue, it's a Wasabi trickery issue, their interface is shit, their retention policy is annoying, their manners were never that helpful when I had the same issue.
Web Design https://www.evergreen.je
Development https://brandlight.org
Life https://marcusquinn.com -
@marcusquinn said in Updates for apps sorted by size:
Seriously, get yourself off of Wasabi, OK service, but sharks for bill shock
Absolutely
I stopped using Wasabi and use Scaleway -
@marcusquinn I get it, totally. And I plan on it.
But the original issue remains.Even If I switch services, without versioning I have nearly 6TB of data being stored.
I want to know where that's from so I can prune as needed.
-
@privsec said in Updates for apps sorted by size:
I want to know where that's from so I can prune as needed.
Again...where do you get the 6TB number from? You can search through your backup files by domain like
app_domain.com*to figure out what app consumes how much within your backup.
So in my case, 25GB compressed appdata ending up as 275GB in my backup. 1 backup per day and 7 days retention. Keep in mind that another another full mirror is stored in the backups "storage" folder under different naming.
Because you have one big app (the light blue one...is it Nextcloud?) with lots of data in it, this could be the reason for your backup size. Also consider using RSYNC if you have lots of huge files that don't change that often.
-
@privsec said in Updates for apps sorted by size:
Even If I switch services, without versioning I have nearly 6TB of data being stored.
I bet you wouldn't.
And as others have explained, just do tarball backups one by one on a new backup provider and then you'll know how much space each app's backup takes up in reality.
-
@privsec said in Updates for apps sorted by size:
This is partly why I want to know what apps have what amount of space in backups.
I have 5.6 tb of data in wasabi, but I have no idea as to why its that high
It does look like my backups in cloudron do not list all the backups in Wasabi
MY thought process is that this is an issue/bug
