Feature Request: Backblaze B2 As Backup Target

[girish]

This is in our todo list.

[moonmeister]

Would love to see this.

[moonmeister]

Backblaze just announced an S3 compatible API for B2. Obviously users can set this up manually but hopefully this makes it faster to include a native integration as well!

https://www.backblaze.com/blog/backblaze-b2-s3-compatible-api/

[girish]

@moonmeister Yup, incredible announcement. They also lacked server side copies (which Cloudron requires)! I will be testing it out.

[girish]

FWIW, you can probably already test this out with "S3 Compatible Provider" in the backups drop-down. If anyone manages to test it out, please let us know here.

[adrw]

Running a full backup test now, will report back results but I'm very excited!

[adrw]

Backup is still running, uploaded >77GB in 7 hours. Would highly recommend TAR backup format since with rsync it blows through the limited Class C Transactions with the Remote Copy and individual file uploads which would end up being pretty costly.

[girish]

This will be part of 5.4

[adrw]

@girish Awesome! This is such exciting new! Thank you for prioritizing it and getting it into the next release.

I'm curious...

1. Will it be using the native B2 API or with their S3 compatible API?
2. Will it be optimized to support rsync backups and not just tar/zip?

Again, huge kudos for baking in B2 support!

[girish]

@adrw It will use the S3 compatible API. Both rsync and zip options will be available.

[marcusquinn]

@adrw Tested using the S3 compatibility setup and it works perfectly with rsync and tarballs, and the rsync actually doesn't cost anything significant so is still worth doing.

Very happy with the backup systems and both Backblaze B2 and Wasabi for speed and value. Great work team Cloudron - and hope that's some confident happiness for everyone wondering if either are worth trying.

[adrw]

@marcusquinn Great to hear! Curious your experience with B2 rsync. I have a fairly large backup (~250GB) and it seems to blast through the Class C transactions very fast to the point of making rsync multiple times more expensive than tar zip backup. I’m giving it another shot based on your experience, hopefully the increased cost is only on first run.

@girish Is there a way I can increase the backup timeout? For my large (~250GB) backup it keeps timing out while uploading ~8mbps to B2 and making 65-70% progress. If the timeout can be configurable or increased it looks like it would succeed. Thanks!

[girish]

@adrw I am working on some fixed for the next release (as part of https://git.cloudron.io/cloudron/box/-/issues/691). Some settings like copy concurrency, upload concurrency, total backup timeout etc seem to be very installation specific. So, we will expose these in the UI.

[adrw]

@girish Great to hear and thanks for prioritizing this. For my own sake, when do you expect those settings will be exposed? I haven't had a successful backup in 3 days since some additional data was uploaded to a Nextcloud instance so I'm starting to sweat a bit. Both tar and rsync fail with the same timeout error after running for hours. If it can land in the next week or two then I'll probably stick it out until then. Otherwise I'm going to revert the data migration so that backups can start again. Many thanks again!

[marcusquinn]

@adrw If ingress is too pricey for your volumes with Backblaze then Wasabi is very competitive and doesn't charge for any ingress up or down.

[adrw]

@marcusquinn Thanks for the recommendation! It only seems to be a bit pricey when using the rsync backup mode. For tar it works great. Both seem to timing out so I'm hoping @girish exposing some more knobs to turn will allow improvement of backup speed.

[adrw]

@girish Good news, rsync backups are now succeeding since the diffs are small and a few failed ones were able to build on each other so no rush on the configuration. Looking forward to testing it out once it lands! Thanks again!

[girish]

@adrw I have scheduled configuring the timing issues for 5.5 in any case. We have a couple of others facing similar issues.

[marcusquinn]

Update on using rsynch to Backblaze B2:

On one Cloudron, I have 23 Apps installed, most are just default installs a few migrated websites.

98GB of backup storage with lifecycle retention of last 2 versions. 1.1 million files.

Backblaze B2 ingress seems to be costing about the same as the storage, last month was $4.95.

The Billing page is shit too as they use the US date-format, the only country in the world with that weirdness and you'd think developers would know better. Maybe their measuring ingress in Pounds, Ounzes, Cups, Quarts, Feet & Inches?

Conclusion being, Backblaze B2 will be cheaper if you're using tarball backups or for up to about 150GB rsync.

Wasabi will be cheaper if you want rsync for incremental backups for more than about 150GB.

Personally I'm leaning towards sticking with rsync for incremental and changing that target to Wasabi as storage needs grow.

Absolute ideal would be to have both; tarball to B2 and rsync to Wasabi - or even have 2 x B2 accounts.

That's all in addition to provider snapshots — so 3 separate physical, jurisdictional and provider locations. Could probably do more with Synchthing as well but that might be overdoing it.

Can't think of a more comprehensive, risk-spread replication and lower cost setup than all that unless anyone has experience otherwise?

[girish]

Wasabi prices are kind of incredible, IMO and I have not seen cheaper options. B2 also has the aspect that download costs money. So if you want to restore 150GB, this costs $1.5.

Only option that is comparable is probably Hetzner storage box. 1TB is around $9. This solution is only practical if your server is somewhere near their storage box.

(Ignoring the cheapest option of buying a external hard disk and attaching it to your physical server I bought a 2TB disk for just $55. The disk will pay for itself in 8 months of cloud storage)