Object Storage or Block Storage for backups of growing 60+ GB?

[scooke]

@d19dotca Assuming Object Storage is the same as S3 or Minio, I would go with Block Storage because I don't always rely solely on the software to manage my backups. Oft-times I want to retrieve a specific file from a backup - combing through that deep and convoluted Object Storage folder hierarchy is insanity-inducing! Also, sometimes a Restore process fails (a Backup one, too) due to one file name's odd character - good luck finding it in that same hierac-hell.

That said, I do use OB for all my major backups. But I also use BS for other more crucial, short-term, files-still-needed backups.

[girish]

One thing to consider is that Block Storage is usually with the same provider (atleast, for "speed"). It's best to have your VPS and backups in separate providers to prevent getting locked out.

Also, my understanding is that most of Block Storage are not RAID and there is no protection against bitrot or redundancy. Object Storage is usually implemented using 'pooled storage' and tends to have redundancy/checksums etc.

[MooCloud_Matt]

@girish
Every storage in a datacenter is manage with some sort of raid/redundancy.

Or you will have too much downtime, due to normal drive failure.

The it depends if they have zfs or a filesystem that have checks and improvements for bitrot preventions.

[girish]

@MooCloud_Matt I see. Info on Block Storage is generally hard to come by.

But for example https://digitalocean.github.io/navigators-guide/book/03-backup/ch07-storage-on-digitalocean.html (I don't know if this is some authoritative documents), it says for Block Storage "Data spans multiple nodes. File systems could experience corruption." It does also say "The Volume storage cluster is a distributed system that has multiple copies of your data within the cluster" . Not sure what all this translates to Does this mean it will protect me against bitrot?

[MooCloud_Matt]

@d19dotca
My advice is not not use block storage because if some one get in to your Cloudron they get in to your backup too.

Set up S3 + your own lifetime policy, and not using incremental backups.
And obviously not allowing the key in Cloudron to delete any thing.

[MooCloud_Matt]

@girish
Do is really bad in explain what they use, even if you contact them.

But mostly likely is just to cover there ass, if you have a distribute system (especially if you use block storage and not file storage) you have parity check in the block of data.

Ceph is the most use currently (if someone can check that, maybe something new is been use) in big Clusters, do all of that as a default.

[robi]

Having worked in the storage industry, there are many insights one gets from years of experience that you cannot get anywhere else.

The little known thing is that stupidity and ignorance always comes first.

Here's the perfect example:
All history of storage has a legacy that started with blocks on spinning disks. Taking this forward in time, all protocols, drivers and interfaces have been written to support block storage. Along comes solid state, and now all the slow block ways of doing things are just put on top of flash storage. Convenient right?

Stupid? You bet.

Now we have speed and volume of data in tsunamis greater than what disks can handle, and along comes object storage, which can do all that and handle sizes greater than terabytes, but petabytes (PB), exabytes (EB) and beyond!

What does the industry do first? They use block storage protocols and write directly to object stores. Convenient right?

Stupid? You bet.

And slow too. Not because object storage is slow, but because using block storage concepts on object stores makes them slow.

It's like putting diesel fuel in your gasoline car. It doesn't match and won't work very well, and may even destroy it.

That is exactly what happens when block storage concepts, with small random writes are used to dump data to a nice object store, who's architecture is optimized for large sequential transactions, for which it's brilliant at, especially at scale. It slow everything down and trashes the system, further causing problems.

Only in the last several years have a few smart folks begun to recognize this difference and write object storage connectors (and even start companies) that are actually tuned and understand what an object store does and use it properly.

Those are how the likes of Apple, Disney and other huge companies with enormous media assets are able to store all the apps, movies and data globally, so things just work when you click "play".

[d19dotca]

Thanks guys! As much as I hate the 1.5 hour backup time (and it'll grow longer as more data is collected), I think it's the safer bet to just do it at like 4 AM or something each day, the storage then is out of the Datacentre and will be safer that way.

Do you guys use rsync or tgz for the object storage backups, do you see any performance improvement using one over the other? I normally used rsync when using block storage as that was usually the quickest, but I'm guessing that's not the case with Object Storage, eh?

[subven]

@d19dotca depends on what kind of data you have. 99% of my backup contains small files below 1mb, no audio/video or big files.

With tgz you get compression especially for data like documents/mails, no problems with naming or deep folder structures and the benefit that you transfer one big file (archive) instead of sometimes a 6 digit number of small files. It takes way longer for RSYNC to check all these files then just to pack and transfer them.

If you have lots of big files that don't compress well, tgz isn't much of a help and transfering them would be waaaaay slower in comparison to RSYNC that just needs to compare source and target.

I use Netcup and Hetzner root server and a 5TB Hetzner Storage Box for tgz backups that does daily snapshots (limited to 3) in addition. For my Hetzner servers I have a storage VPS with Minio at AlphaVPS.

While having your backup storage at the same provider has speed benefits, it is bad practice and destroys what you actually want to achieve with a backup in the first place.

[d19dotca]

@subven Okay I ran some tests. If using tgz it was about 1.5 hour upload. If I used rsync, the first one was about an hour (I think it was 55 minutes), but the second and third one so far have been in the area of 22 minutes, so much quicker. I guess the downside is this takes up more storage than tgz would have but that's okay I guess since Wasabi takes the full 5.99 USD for 1 TB regardless of how much is stored under 1 TB. I guess I'll continue to use Wasabi via Cloudron's rsync type for now.

[MooCloud_Matt]

@d19dotca
Please check better wasabi pricing, is 5.99 for stored and deleted files.
With rsync Cloudron will create a lot of delete request, and you will end up paying a lot.

[MooCloud_Matt]

@d19dotca said in Object Storage or Block Storage for backups of growing 60+ GB?:

more storage

Depends if you store backup for 2 days or for 1 year.
Rsync is an incremental backup, you will never store the same file 2 times.
But is disadvantage is that I'd you even get 1 corrupted snapshot you will lose everything after that.

[d19dotca]

@MooCloud_Matt Would you suggest Backblaze over Wasabi in that case? Backblaze charges on command types too so I assumed I’d be better with Wasabi but point taken as their 3-month lifespan requirement struck me as very strange. Went with Wasabi still as it seemed unlikely I’d hit the 1TB limit even with deleted files, plus it has a Canadian Datacentre which will be more performant latency-wise then Backblaze’s California Datacentre when my VPS is hosted in Toronto, Canada.

[LoudLemur]

@d19dotca On Vultr, object storage starts at $5/month for 250GB and 1TB of transfer. Block storage is $25/month for 250GB.

[robi]

@LoudLemur Seems to be half that price on Contabo, even less on Wasabi.

[LoudLemur]

@robi said in Object Storage or Block Storage for backups of growing 60+ GB?:

@LoudLemur Seems to be half that price on Contabo.

I was surprised as the Vultr Object Storage is nvme, but the block storage can be HDD, so I thought the block storage would be cheaper.

What is the best toolto browse through block storage files to find one you want?

[robi]

@LoudLemur there are many tools, depends what OS you use.. WinSCP or say Cyberduck on MacOS.

[d19dotca]

@LoudLemur Yeah I used Vultr for a little bit when testing but their Datacentre location for their object storage was far away from my VPS Datacentre in Vultr, and it's not too cheap at least compared to what sort of storage you get with Wasabi or Backblaze for example. Not too bad though, for sure, and worth consideration for some.

[LoudLemur]

@d19dotca said in Object Storage or Block Storage for backups of growing 60+ GB?:

@LoudLemur Yeah I used Vultr for a little bit when testing but their Datacentre location for their object storage was far away from my VPS Datacentre in Vultr, and it's not too cheap at least compared to what sort of storage you get with Wasabi or Backblaze for example. Not too bad though, for sure, and worth consideration for some.

[MooCloud_Matt]

@d19dotca
You must understand what is more valuable for you: speed, reliability, or price.
reliability mostly Backblaze’s services are one of the best.
Speed, Wasabi is good enough if is near to your data center, but the is no duplication of data, and is something that we have to use more than want I like to admit.

[d19dotca]

@LoudLemur & @MooCloud_Matt

Interesting. I decided to test something with a local Datacentre closer to my VPS called IDrive e2 (seems like a recent s3 competitor from mid-2022 which promises high speeds).

Backblaze is good although I find it quite slow (mostly because my VPS is in a very far away Datacentre from Backblaze's California (us-west) location. Speed isn't critical since it's just backups but definitely helps still.

Backblaze's pricing for their API calls scares me a little bit, makes me think it'll be much more pricey than I'm anticipating. May just need to test it out for a while to verify.

I see what you mean about Wasabi's weird 90-day storage policy which means even deleted files are still counted for 90 days, and my current estimate is quickly adding up, so I think despite initially happy with Wasabi's performance I may need to abandon that provider.

Still experimenting. Currently in the middle of a large 60+GB backup to IDrive e2 (using rsync instead of tarball for now) and have to say I'm super impressed with the speeds. Their pricing is also quite minimal. Will see if they end up being the one I use.

[LoudLemur]

@MooCloud_Matt said in Object Storage or Block Storage for backups of growing 60+ GB?:

@d19dotca
You must understand what is more valuable for you: speed, reliability, or price.
reliability mostly Backblaze’s services are one of the best.
Speed, Wasabi is good enough if is near to your data center, but the is no duplication of data, and is something that we have to use more than want I like to admit.

And price? Which one?

[MooCloud_Matt]

@LoudLemur probably contabo, they use ceph.
This means that u have replication by default and a really good support for S3.

[d19dotca]

FWIW, I’m quite impressed with the pricing and performance of the newer IDrive e2 storage (with s3 API).

Curious though on a related but different topic… when uploading to s3, do you find yourself using rsync for larger backups or do you opt to just use tgz? I so far tend to find rsync a bit more performant (likely because of the concurrency settings) but the downside is it takes forever to delete files out of the bucket when there’s so many of them, the deletion process even with s3 API calls is incredibly slow when so many files exist. Makes me think it may be better to just stick to tarball images instead.

[doodlemania2]

@d19dotca do you know much about these idrive folks / would you recommend? the pricing almost makes them look like a scam - 2TB of storage for $8/year unlimited egress? i mean, they LOOK legit, but wow that's a helluva deal.

[d19dotca]

@doodlemania2 IDrive has been around for many years for the computer backup solution (competitors to Backblaze and Carbonate for example). This year they released their s3 storage competitor, so they're basically following in the same shoes as Backblaze. It's all legitimate. They're newer so they're trying to sweeten the deal to attract people away from Backblaze and such, hence the lower pricing if you buy their promo for the year plan. It's only that price for the first year though, definitely not a scam IMO.

I'm currently just trialling with their monthly plan as it's only $0.004 USD per GB, so to store even 800 GB of backups in a month would only be $3.20 USD.

[doodlemania2]

@d19dotca I keep thinking that between these newcomers with their free egress and cloudflare shaming them, that Azure and AWS et al will drop the egress fees (or at least lower them)!

[LoudLemur]

@doodlemania2 Egress fees? If you want to take your data away, you have to pay?

[scooke]

@d19dotca You might not believe it, but my idrive account goes back to 2009!

[doodlemania2]

@scooke funny story - i just migrated all my stuff to idrive for my mastodon instance only to discover - twoops - they don't appear to allow public buckets!?

[scooke]

@doodlemania2 I also signed up (the regular idrive.com account didn't work with the newer idrive e2), activated S3, made a bucket, checked out the user info, and then saw the buckets were private (for now). Bad timing, hey!

[robi]

@doodlemania2 I think that's a setting before you create the bucket.

[scooke]

@robi Unfortunately it is so. Not sure when they will enable the setting.

[doodlemania2]

@scooke anyone know a contact? Would like to stay with them but no public buckets...lol - maybe they can bit flip it for me?

[doodlemania2]

went with their main tech support email

[robi]

@scooke all regions?

[d19dotca]

Noticing that tgz of about 35-40 GB GB (well that’s the size after compression, it’s really compressing closer to 60+ GB uncompressed data) and it’s nearly double what it consumes in storage because at least the first backup has that snapshot directory too which is a duplicate of the latest backup. My IDrive e2 bucket shows a total usage of 70 GB currently after one system backup.

I notice if I use rsync the backup times are anywheres between about 20-70 minutes, however when using tgz it’s closer to 120 minutes. That’s about the same performance when using Wasabi and even better than Backblaze in my tests so far.

The downside to rsync is it takes forever to delete so many (tens of thousands of) different files from s3 storage, which means Cloudron cleaning it up or even just me doing it manually to purge older data is a real time sink. Makes me want to use tarballs instead however the upload rate seems to be much less performant. I suppose it is maybe a moot point to some degree because it’d normally be done in the middle of the night when time isn’t really a factor, although it will mess things up for two hours during Cloudron version updates if I (which I would) do system backup prior to that task as the maintenance would be so much longer.

I’m curious though… is there a way to improve the tarball performance? Why is it so dang slow? I assume concurrency but what prevents concurrency from taking place when using tarball? @staff, any suggestions here? Anything I can do or any specific recommendations?

I should note I’ve seen next to zero improvements or changes at all whether I use a part size of 128 MB or 1 GB, doesn’t seem to change anything performance wise.

[girish]

@d19dotca said in Object Storage or Block Storage for backups of growing 60+ GB?:

I’m curious though… is there a way to improve the tarball performance? Why is it so dang slow? I assume concurrency but what prevents concurrency from taking place when using tarball? @staff, any suggestions here? Anything I can do or any specific recommendations?

One idea might be to measure just uploading via other tools to see how much the performance difference is. While the tgz code uploader is slow, it's not substantially slower than native tools (from what I have tested). Try to do a tar cvf - upload of the same data directory as the app to the backup storage and measure.

[scooke]

@robi I checked most (all non-US ones, 4 US ones) and the Public radio button stayed greyed out.

[robi]

@scooke let's see what support says.. unless they have a status page where one can see a portion of their services being offline (which provides this public access to a private bucket).

[doodlemania2]

@scooke So I heard back from email based support - they turned public buckets on for my account!

[marcusquinn]

@d19dotca Thanks for your R&D sharing on this, just trialing it, and the 1 year deals save a lot of money compared to any other S3, including Wasabi.

Really like the UI/UK, much easier for anyone to work with than Wasabi too, as you don't need to create Policies in JSON like you do with other.

from me, definitely recommended for all home users, and most SME needs too!

@d19dotca said in Object Storage or Block Storage for backups of growing 60+ GB?:

Interesting. I decided to test something with a local Datacentre closer to my VPS called IDrive e2 (seems like a recent s3 competitor from mid-2022 which promises high speeds).

[d19dotca]

FYI, got an email today from IDrive which announced the public buckets for people looking for that, mentioning it here as I know a few people were asking about that feature.

They’ve also made some serious performance tweaks. I noticed before I was seeing it take about 55-75 minutes for a 30+ GB upload in tarball, and now it’s closer to 45-50 minutes for the same size (in fact possibly even larger of a file now to boot as I have a couple more apps deployed now too and email continues to grow larger).

[robi]

@d19dotca said in Object Storage or Block Storage for backups of growing 60+ GB?:

I noticed before I was seeing it take about 55-75 minutes for a 30+ GB upload in tarball, and now it’s closer to 45-50 minutes for the same

It may be related to lower traffic and usage over the end of year holidays.

[d19dotca]

@robi said in Object Storage or Block Storage for backups of growing 60+ GB?:

It may be related to lower traffic and usage over the end of year holidays.

Entirely possible although I was discussing with them in a support case about the general speed of things and they did say they had recently implemented a change to their service which should speed things up and in my experience it does seem to be improved.

[doodlemania2]

@d19dotca I also noticed an increase in performance. One thing I did notice though - my public buckets were part of the beta and included anonymous root "viewing" of sorts - it generated an XML file of the contents of each public bucket. A quick email to support and they had me turn the bucket to private, then back to public to correct. Something to check if you or someone you love may be an early public bucket adopter there!

[marcusquinn]

@d19dotca We're noticing some significant slowdowns with IDrive backup uploads now. (We have lots and lots of small files and over 1TB). Just wondering if you'd experienced anything that might seem like throttling?

[scooke]

@marcusquinn Oh oh... too good to be true?

[marcusquinn]

@scooke Maybe, trying their support.

[MooCloud_Matt]

@marcusquinn
idk what tech is behind their stack, but if it is just HDD with no NVMe for caching metadata or any kind of index.
100% that it will be slow if they grow too much.

[marcusquinn]

Changing strategy to Tarball and that seems to complete in a few minutes. Might just have to be the trade-off for our apps, that have lots and lots of small files. More S3 storage usage but faster to backup and restore, and more self-contained for each backup not relying on files from others.

[robi]

@marcusquinn yes, that is the more optimal format for object storage, plus incremental diffs.

We discussed elsewhere on the forum a hybrid option inbetween rsync by file and large tgz.

[marcusquinn]

Yeah, I'm gonna call this, any backups >100GB or >100,000 files is likely to get impractically slower for most budget S3 storage. Might be worth a note/tooltip in the settings to suggest tarball for Cloudron servers over these numbers.

TBH I think the compression and minimal numbers of files being uploaded, with a sensible retention policy, is going to offset any storage-saving from using rsync. Rsync is a nice idea in theory for smaller directories, but I feel the file count costs are higher than storage costs for local compression and uploading that for each backup run.

[d19dotca]

@marcusquinn I haven’t noticed that myself but been using tgz the whole time and it’s been quite fast overall so far.

[marcusquinn]

To confirm on my R&D inspired by this thread. We're happy now with:

• IDrive on the introduction offer pricing
• Tarball for the backup method
• IDrive and Tarball encryption
• 7 Daily, 4 Weekly, 12 Monthly retention policy