[Backups] Ability to add multiple storage provider/location

[p44]

Should be interesting to add multiple storage provider/location and give ability to have additional backups on different locations.

Hope request was clear

[ruihildt]

Would be nice to have, to make sure we follow the rule of having multiple backups located in different places.

[robi]

I suggested this before here

Very useful for more than one use case.

[p44]

@robi Robi I didn't see at all your post and is interesting that both considered to add that feature.

If @girish want can delete this post.

[girish]

@p44 It's fine. I think the other one is more like a "download backup" feature. This one is for storing backups to multiple locations. I want this but at the same time don't want to overcomplicate our backup system. The 3-2-1 rule of backups is to store it in 2 different locations, so we should encourage this.

[girish]

To add to this, for home installations, I want to be able to push encrypted backups to a friend's cloudron (after he/she has given me permission, of course).

[robi]

@girish yes or to places like keybase, gdrive, onedrive, etc..

[edapm]

@girish Yes, I would be good to say have a Minio bucket to which to push backups to on your friends' Cloudrons, and be able to have a backup on each Cloudron. (See diagram below)

*Small square is Minio bucket

[marcusquinn]

It might be more CPU/bandwidth friendly if one Rsync process sends the backup to location A, as it does right now, and a second initiates the copy from B to C, and if desired a D, E, F location too if people wanted to go crazy with backup locations.

From another angle, it might be that we want one entirely different backup method to one place (could be unencrypted rsync), and another to another (perhaps encrypted tarball).

Would very much like this though, especially as cyber attacks are only growing as more and more economic value is online, and on other people's servers.

[girish]

@marcusquinn yes, we will try to do this for the release after. I think it's important to support 3-2-1 style backups - atleast the 3-2 part more easily.

[MooCloud_Matt]

A good way to handle the 3-2-1 for cloudron can be to replicate what Proxmox has done, delegate the replication to the other software(Proxmox Backup Server) installed on the destination backup server, this is for 2 reasons: storage servers/vps often have low resources but those are good enough to enable replication and they are really cheap, probably more expensive then Wasabi, but Wasabi is slow and really not efficient in cost for cold Rsync backup.

[MooCloud_Matt]

I was looking around for a way to solve this issue without using a complicated setup like Ceph (like we did, and we would love to get out of it but that's another story).
I found Restic, it could be used instead of rsync, and it supports S3 but also their own rest server that can run on the destination server of the backup and with their cli, you can create a copy of one of the snapshots without involving the production server, and store it on an HDD offline or on the others providers.

https://github.com/restic
https://github.com/restic/rest-server

[jdaviescoates]

@moocloud_matt said in [Backups] Ability to add multiple storage provider/location:

Restic

There's a bunch of discussion about that here:

https://forum.cloudron.io/post/2466

Sounds like both @necrevistonnezr and @fbartels have some experience with it

[fbartels]

@jdaviescoates yes, restic is quite nice. I am however not using their server backend, but rather push backups to a s3 target.

I am however not sure how restic would solve the 3rd (offsite) part. I kind of achieve this by mirroring s3 buckets, but this is something I can already do when Cloudron is already writing stuff to s3.

[MooCloud_Matt]

@fbartels
you have a command/API that can copy a snapshot to a new location.
(i just had time to check their documentation and a basic install)

[fbartels]

@moocloud_matt ah, true. This however means that the copy needs to go through a local client (download & upload) and apparently also reencrypts data (with the potential of deduplication not working).

https://restic.readthedocs.io/en/latest/045_working_with_repos.html#copying-snapshots-between-repositories

Its been ages since I last looked into restic, as it "just worked" for me. The thing to highlight is that apparently there has been a successful handover in maintainership in the past, as the original author hasn't done any work himself in quite a while.

Another upside is its portability due to being written in golang.

Edit: if one wants to look into restic, https://autorestic.vercel.app/ is a nice wrapper to simplify setup and handling. My old systems still use bits of bash for that.

[MooCloud_Matt]

@fbartels said in [Backups] Ability to add multiple storage provider/location:

local client (download & upload)

True but is Go, and a Storage Server/VPS have enough resources to have both server and client on it to replicate to an cold s3 storage.

@fbartels said in [Backups] Ability to add multiple storage provider/location:

Another upside is its portability due to being written in golang.

We also need to take into consideration that setup a golang script is easy and can be done by almost anybody in the cloudron community and tanks to the Rest API, cloudron can manage the server super easily, instead if it will be selected a nodejs or py script it can be a lot more difficult to setup.
I think that docker should be excluded as a tool to distribute the destination server for the backup, because many VPS provider of Storage Server offers OpenVZ and not KVM, we would not have that issue but many the will.

[robi]

Don't forget about rclone.org

[MooCloud_Matt]

@robi
I actually don't like that the 3-2-1 is managed by the main server, because if that is compromised you will have compromised also your backup.

I think that: if cloudron wants to offer a better backup solution should have a 3° party software/node be in charge of the replication for the 2-1.
This will protect the server from any ransomware or if your server is compromised.

[marcusquinn]

Borrowing an answer from StackOverflow that may work:

• Minio Cloudron instance: using the command mc mirror on a cron job.

If that works, it could just be a case of documenting and maybe a GUI to make it user-friendly @girish ?

[MooCloud_Matt]

@marcusquinn
a full install of cloudron is too many resources w8st for many storage servers, we speak of old CPU (many of our storage servers have Haswell xeon) or just 1 vCore (time4vps) , and often without the support of docker.

[robi]

@moocloud_matt said in [Backups] Ability to add multiple storage provider/location:

I actually don't like that the 3-2-1 is managed by the main server, because if that is compromised you will have compromised also your backup.

That's the problem with traditional backups.

Next gen way of thinking about backups is simply having a much more resilient storage system. For example, when your data is sprinkled across 8 places and you only need 5 to restore any file/object. There are some very clever and efficient algorithms for this m of n approach which removes the need for 3x replication.

Minio can do this, and as a community we can pool resources to have 20+ places and only need 7 or so to be available at any one time. Maybe even start a coop.

[neurokrish]

@robi great idea. +1 for m of n approach with minio!

[MooCloud_Matt]

@robi said in [Backups] Ability to add multiple storage provider/location:

For example, when your data is sprinkled across 8 places and you only need 5 to restore any file/object.

True, that's why we are using ceph, but it's not efficient (storage speaking) to protect the files we need to use snapshot or versioning in ceph too, because if the access of the bucket is compromised on the cloudron side all files even if they are slit into multiple nodes can still be deleted/encrypted, so that made the all advantage of using Software Define Storage/ Distributed storage = to a normal NAS offer by the datacenter over NFS.

I really would like to analyze better what push proxmox to build a dedicated Client for their Storage Server. And what I have understood un till now is that they want maximal protection made easy which means that the ssh key used by their hypervisor server is not able to access the 2 and 1 copy of the backup.

I really don't care about what software/ stack Cloudron will use, I just want to get out of Ceph for the backup and use a better setup that is not less safe.

[marcusquinn]

@robi Kinda what IPFS is?

• https://forum.cloudron.io/topic/6280/ipfs-desktop-cluster-peer-to-peer-hypermedia-file-sharing

[robi]

@moocloud_matt proper object storage will have versioning and dedup too.

If you have source compromise you have other issues, however aspects of the push to store can be flipped in a pull to archive manner provided you have a mechanism to detect compromise and stop the pull or avoid overwrites.

[MooCloud_Matt]

@robi

@robi said in [Backups] Ability to add multiple storage provider/location:

will have versioning and dedup too

True, but dedup is a "raid" equivalent feature but scaled to multi-node.
And versioning is not efficient if combined with backup tools, because most of them don't take advantage of s3 versioning API.

So we are back on the idea that the main server if it's is compromised you want your backup to be safe, so many ransomware those days.

Yes, that's possible to do it with S3 but is not easy and requires a lot of coding from the cloudron side and is just 1 protocol, no sftp or nfs or samba, if you don't want to use a proxy s3 to nfs or like we use Ceph to smd.

Supporting 1 protocol is not what backups are in cloudron, you see more and more protocol support over time, I just help add SMB3 (with encrypt) to that list.

And I think that is amazing for cloudron users that are not geeks or professionals and want to keep their local Synology or qnap NAS as a backup.

A solution that would fit users like many that are commenting on this post is not enough. The idea I hope is to find a solution that can scale from a classic rsync stile to a 3-2-1 systems like proxmox backup or Borg.

[robi]

@moocloud_matt for your Ceph replacment you should look at Cleversafe from IBM. They also have some partners that make it even better on the user side. Komprise comes to mind.

For Cloudron, a nice rclone integration would handle most of those options nicely.

[d19dotca]

Definitely having a need for this now as I look to further improve my DR plan. What I’m hoping to achieve… faster local backups to an external block storage disk daily (possibly even multiple times given criticality of email for some users), and perhaps once a week or every 2-5 days would be an external backup to something like Wasabi to avoid any complete disasters in a single datacentre.

[girish]

Yes, this is high on our list. Just finishing up the backup async rewrite! Almost there with 7.2.