Performance of Object Storage, any insights?

[d19dotca]

Hey everyone,

I’m looking at moving my public cloud instance to a VPS with a different company in Canada, but the downside is they only run VPS and don’t support the concept of external disks / block storage, which dampens my backup workflow of using file system on an external disk for high performance.

This got me thinking again about Object Storage providers, like Linode, DigitalOcean, OVH, etc and the performance associated with it.

Anyone have experience using different object storage providers and notice any performance differences between them? Any recommendations based on performance?

Why performance is important here… I have a lot of data to backup and when trying with OVH Object Storage it ends up taking well over an hour to complete. I really don’t want the backup process running that long if I can help it, plus it becomes extra annoying when doing any upgrades which requires (best practice) a backup prior. What should be a ten minute upgrade process for Cloudron for example becomes 1-2 hours long when Object Storage is slow and lots of data to backup.

Any suggestions? Just curious what people’s experience tells them when it comes to Object Storage performance.

I have also considered just running a separate VPS entirely to backup to, but it’s a costly thing to do that at the moment. I also even pondered if I just had a large disk on the main VPS if I could use the same file system but I know that’s definitely not recommended and not a great idea in case something happens with the physical server my VPS would reside on. Of course they should have their own backups but I can’t rely on that.

[robi]

I used to be an Object Storage expert and spent time working on the technology at IBM, where I wrote a Redbook on it.

You need to understand that you cannot treat objects like files.

File access patterns are designed for disks with sectors and blocks; 4k bytes.

Object access patterns are designed for large chunks; 100s of MBs to TBs & PBs at times. Read, they don't like a lot of small transactions.

I've been using Scaleway for backups until they screwed up their systems in FR.
Then added the IBM Cloud Object Storage (COS) system to the mix and it is much more performant.

See the other threads about rclone and mounting remote object stores.

[d19dotca]

@robi Would the takeaway here be then that Object Storage - while it works - isn't exactly the best method to use for backups in Cloudron? Guess it depends a bit on what's being hosted on it though too.

[robi]

@d19dotca no, it's fine for backups, but to fully optimize it, we need to treat it like object storage, not file storage.

That requires a rethink on how we handle, bundle and send out backup files.

For example, avoid rsync mode, tons of small writes. tgz is better, but we can do even better.

[d19dotca]

@robi Ah okay, makes sense. That kind of jives with my experience too where tgz was much better than rsync, but even tgz was still slow.

[robi]

@d19dotca yeah. It would be good to extract my other comments into this thread for clarity.. if you have time.

[girish]

One thing is when an app is backing up, the app itself is not down. It's still accessible and usable by users. So, even if it takes a bit long, I understand it's annoying but atleast there is no real downtime. Only when the container is re-created (which usually takes under a minute) is there a downtime.

To answer the post itself, best to choose an object storage provider/region which is geographically close to the server. Other than that, I have not done any specific performance measurements.

[d19dotca]

@girish Fair enough. I’m trying with Vultr Object Storage at the moment (just started some tests today) and it’s performance is a smidge better than OVH so far with the same general settings, getting around 8-15 Mbps for most of the app updates, but one thing I noticed behaviour for both OVH and Vultr (so I assume others too)… when it gets to the box data (emails) it slows to a crawl and ranges from just 1-3 Mbps, which when uploading about 25 GB of box data makes this an incredibly lengthy process. Any ideas why that might be consistently dropping when Cloudron gets to the box data?

[girish]

@d19dotca not enirely sure, but an idea is to maybe test upload speed via the aws cli tools.

[robi]

@d19dotca said in Performance of Object Storage, any insights?:

when it gets to the box data (emails) it slows to a crawl

If the backup is processing emails individually, that would make sense. Too many small files, makes the object store communication too chatty slowing down actual data transfer.

If it's a bigger one lump tgz it will be faster.

[d19dotca]

@robi I've always been using the tgz format because it always seemed more performant. I get the same behaviour though pretty much no matter what format I use, I noticed, as I just tried with rsync and basically the same thing in the logs showing the transfer speed really slow during the boxdata backup portion, but like 3 times faster on the other portions of the backup.

Funny enough, in my experience rsync was always really bad for Object Storage doing extensive testing with OVH Object Storage. I tried rsync just a bit ago today, the first full one and then a copy. The first full one took a long time (about 1.25 hours), but the subsequent one took only about 17 minutes which I was super impressed with. I never once got that behaviour in OVH where the second rsync time, it was always well over an hour no matter what. Perhaps Vultr's Object Storage is better though with the copy commands or something than OVH was.

[robi]

@d19dotca well the backup is complex.. and doesn't do a full backup every time.. it does deltas of the snapshot of the previous one. So it depends entirely on how many changes happened since then.

[d19dotca]

@robi Oh yeah I get that, that's how rsync works, but I mean at OVH I never got the "rsync benefit" of differential backups when using it, and my server isn't busy enough to have thousands of files changed in a few hours, lol, but yet it seems to be working where I can see the benefit of rsync when using Vultr's Object Storage. I know that OVH uses OpenStack in the backend which also used something it auto-created called "+segments" folders, so it could have been some weird compatibility thing between rsync and OVH's Object Storage / OpenStack.

[robi]

@d19dotca Swift based object stores are one of the least advanced.

IBMs is one of the most. (20GB free)