Object Storage or Block Storage for backups of growing 60+ GB?
-
FYI, got an email today from IDrive which announced the public buckets for people looking for that, mentioning it here as I know a few people were asking about that feature.
They’ve also made some serious performance tweaks. I noticed before I was seeing it take about 55-75 minutes for a 30+ GB upload in tarball, and now it’s closer to 45-50 minutes for the same size (in fact possibly even larger of a file now to boot as I have a couple more apps deployed now too and email continues to grow larger).
--
Dustin Dauncey
www.d19.ca -
FYI, got an email today from IDrive which announced the public buckets for people looking for that, mentioning it here as I know a few people were asking about that feature.
They’ve also made some serious performance tweaks. I noticed before I was seeing it take about 55-75 minutes for a 30+ GB upload in tarball, and now it’s closer to 45-50 minutes for the same size (in fact possibly even larger of a file now to boot as I have a couple more apps deployed now too and email continues to grow larger).
@d19dotca said in Object Storage or Block Storage for backups of growing 60+ GB?:
I noticed before I was seeing it take about 55-75 minutes for a 30+ GB upload in tarball, and now it’s closer to 45-50 minutes for the same
It may be related to lower traffic and usage over the end of year holidays.
-
@d19dotca said in Object Storage or Block Storage for backups of growing 60+ GB?:
I noticed before I was seeing it take about 55-75 minutes for a 30+ GB upload in tarball, and now it’s closer to 45-50 minutes for the same
It may be related to lower traffic and usage over the end of year holidays.
@robi said in Object Storage or Block Storage for backups of growing 60+ GB?:
It may be related to lower traffic and usage over the end of year holidays.
Entirely possible although I was discussing with them in a support case about the general speed of things and they did say they had recently implemented a change to their service which should speed things up and in my experience it does seem to be improved.
--
Dustin Dauncey
www.d19.ca -
@robi said in Object Storage or Block Storage for backups of growing 60+ GB?:
It may be related to lower traffic and usage over the end of year holidays.
Entirely possible although I was discussing with them in a support case about the general speed of things and they did say they had recently implemented a change to their service which should speed things up and in my experience it does seem to be improved.
@d19dotca I also noticed an increase in performance. One thing I did notice though - my public buckets were part of the beta and included anonymous root "viewing" of sorts - it generated an XML file of the contents of each public bucket. A quick email to support and they had me turn the bucket to private, then back to public to correct. Something to check if you or someone you love may be an early public bucket adopter there!
-
@d19dotca We're noticing some significant slowdowns with IDrive backup uploads now. (We have lots and lots of small files and over 1TB). Just wondering if you'd experienced anything that might seem like throttling?
Web Design https://www.evergreen.je
Development https://brandlight.org
Life https://marcusquinn.com -
@d19dotca We're noticing some significant slowdowns with IDrive backup uploads now. (We have lots and lots of small files and over 1TB). Just wondering if you'd experienced anything that might seem like throttling?
-
@marcusquinn Oh oh... too good to be true?
@scooke Maybe, trying their support.
-
@d19dotca We're noticing some significant slowdowns with IDrive backup uploads now. (We have lots and lots of small files and over 1TB). Just wondering if you'd experienced anything that might seem like throttling?
@marcusquinn
idk what tech is behind their stack, but if it is just HDD with no NVMe for caching metadata or any kind of index.
100% that it will be slow if they grow too much. -
Changing strategy to Tarball and that seems to complete in a few minutes. Might just have to be the trade-off for our apps, that have lots and lots of small files. More S3 storage usage but faster to backup and restore, and more self-contained for each backup not relying on files from others.
Web Design https://www.evergreen.je
Development https://brandlight.org
Life https://marcusquinn.com -
Changing strategy to Tarball and that seems to complete in a few minutes. Might just have to be the trade-off for our apps, that have lots and lots of small files. More S3 storage usage but faster to backup and restore, and more self-contained for each backup not relying on files from others.
@marcusquinn yes, that is the more optimal format for object storage, plus incremental diffs.
We discussed elsewhere on the forum a hybrid option inbetween rsync by file and large tgz.
-
Yeah, I'm gonna call this, any backups >100GB or >100,000 files is likely to get impractically slower for most budget S3 storage. Might be worth a note/tooltip in the settings to suggest tarball for Cloudron servers over these numbers.
TBH I think the compression and minimal numbers of files being uploaded, with a sensible retention policy, is going to offset any storage-saving from using rsync. Rsync is a nice idea in theory for smaller directories, but I feel the file count costs are higher than storage costs for local compression and uploading that for each backup run.
Web Design https://www.evergreen.je
Development https://brandlight.org
Life https://marcusquinn.com -
Yeah, I'm gonna call this, any backups >100GB or >100,000 files is likely to get impractically slower for most budget S3 storage. Might be worth a note/tooltip in the settings to suggest tarball for Cloudron servers over these numbers.
TBH I think the compression and minimal numbers of files being uploaded, with a sensible retention policy, is going to offset any storage-saving from using rsync. Rsync is a nice idea in theory for smaller directories, but I feel the file count costs are higher than storage costs for local compression and uploading that for each backup run.
@marcusquinn I haven’t noticed that myself but been using tgz the whole time and it’s been quite fast overall so far.
-
To confirm on my R&D inspired by this thread. We're happy now with:
- IDrive on the introduction offer pricing
- Tarball for the backup method
- IDrive and Tarball encryption
- 7 Daily, 4 Weekly, 12 Monthly retention policy
-
To confirm on my R&D inspired by this thread. We're happy now with:
- IDrive on the introduction offer pricing
- Tarball for the backup method
- IDrive and Tarball encryption
- 7 Daily, 4 Weekly, 12 Monthly retention policy
@marcusquinn how happy are you with idrivee2 these days? I started a trial on Thursday and all was working well, but since Friday I am getting errors left right and center. Opened a support call and it's with their back end team. I am waiting for a response from them, hopefully Monday.
-
@marcusquinn how happy are you with idrivee2 these days? I started a trial on Thursday and all was working well, but since Friday I am getting errors left right and center. Opened a support call and it's with their back end team. I am waiting for a response from them, hopefully Monday.
@vansens Still seems to be working. You followed the above settings? (I doubt Rsync will work)
-
@vansens Still seems to be working. You followed the above settings? (I doubt Rsync will work)
@marcusquinn I found it was an issue with idrivee2. Moved bucket and backups to another region and all started to work again
-
Interesting. I decided to test something with a local Datacentre closer to my VPS called IDrive e2 (seems like a recent s3 competitor from mid-2022 which promises high speeds).
Backblaze is good although I find it quite slow (mostly because my VPS is in a very far away Datacentre from Backblaze's California (us-west) location. Speed isn't critical since it's just backups but definitely helps still.
Backblaze's pricing for their API calls scares me a little bit, makes me think it'll be much more pricey than I'm anticipating. May just need to test it out for a while to verify.
I see what you mean about Wasabi's weird 90-day storage policy which means even deleted files are still counted for 90 days, and my current estimate is quickly adding up, so I think despite initially happy with Wasabi's performance I may need to abandon that provider.
Still experimenting. Currently in the middle of a large 60+GB backup to IDrive e2 (using rsync instead of tarball for now) and have to say I'm super impressed with the speeds. Their pricing is also quite minimal. Will see if they end up being the one I use.
@d19dotca said in Object Storage or Block Storage for backups of growing 60+ GB?:
Still experimenting. Currently in the middle of a large 60+GB backup to IDrive e2 (using rsync instead of tarball for now) and have to say I'm super impressed with the speeds.
Have you managed to connect idrive to your Cloudron as a volume for storage? How did you manage to do that? Can you do it from within the Cloudron panel?
-
@d19dotca said in Object Storage or Block Storage for backups of growing 60+ GB?:
Still experimenting. Currently in the middle of a large 60+GB backup to IDrive e2 (using rsync instead of tarball for now) and have to say I'm super impressed with the speeds.
Have you managed to connect idrive to your Cloudron as a volume for storage? How did you manage to do that? Can you do it from within the Cloudron panel?
@LoudLemur I find rsync too slow with zilions of small files. You'd think it would be quicker as incremental, but large compressed files are beating many small files, same for storage space, compressed significantly less storage costs.
