[Backups] Ability to add multiple storage provider/location

[p44]

@robi Robi I didn't see at all your post and is interesting that both considered to add that feature.

If @girish want can delete this post.

[girish]

@p44 It's fine. I think the other one is more like a "download backup" feature. This one is for storing backups to multiple locations. I want this but at the same time don't want to overcomplicate our backup system. The 3-2-1 rule of backups is to store it in 2 different locations, so we should encourage this.

[girish]

To add to this, for home installations, I want to be able to push encrypted backups to a friend's cloudron (after he/she has given me permission, of course).

[robi]

@girish yes or to places like keybase, gdrive, onedrive, etc..

[A Former User]

@girish Yes, I would be good to say have a Minio bucket to which to push backups to on your friends' Cloudrons, and be able to have a backup on each Cloudron. (See diagram below)

*Small square is Minio bucket

[marcusquinn]

It might be more CPU/bandwidth friendly if one Rsync process sends the backup to location A, as it does right now, and a second initiates the copy from B to C, and if desired a D, E, F location too if people wanted to go crazy with backup locations.

From another angle, it might be that we want one entirely different backup method to one place (could be unencrypted rsync), and another to another (perhaps encrypted tarball).

Would very much like this though, especially as cyber attacks are only growing as more and more economic value is online, and on other people's servers.

[girish]

@marcusquinn yes, we will try to do this for the release after. I think it's important to support 3-2-1 style backups - atleast the 3-2 part more easily.

[MooCloud_Matt]

A good way to handle the 3-2-1 for cloudron can be to replicate what Proxmox has done, delegate the replication to the other software(Proxmox Backup Server) installed on the destination backup server, this is for 2 reasons: storage servers/vps often have low resources but those are good enough to enable replication and they are really cheap, probably more expensive then Wasabi, but Wasabi is slow and really not efficient in cost for cold Rsync backup.

[MooCloud_Matt]

I was looking around for a way to solve this issue without using a complicated setup like Ceph (like we did, and we would love to get out of it but that's another story).
I found Restic, it could be used instead of rsync, and it supports S3 but also their own rest server that can run on the destination server of the backup and with their cli, you can create a copy of one of the snapshots without involving the production server, and store it on an HDD offline or on the others providers.

https://github.com/restic
https://github.com/restic/rest-server

[jdaviescoates]

@moocloud_matt said in [Backups] Ability to add multiple storage provider/location:

Restic

There's a bunch of discussion about that here:

https://forum.cloudron.io/post/2466

Sounds like both @necrevistonnezr and @fbartels have some experience with it

[fbartels]

@jdaviescoates yes, restic is quite nice. I am however not using their server backend, but rather push backups to a s3 target.

I am however not sure how restic would solve the 3rd (offsite) part. I kind of achieve this by mirroring s3 buckets, but this is something I can already do when Cloudron is already writing stuff to s3.

[MooCloud_Matt]

@fbartels
you have a command/API that can copy a snapshot to a new location.
(i just had time to check their documentation and a basic install)

[fbartels]

@moocloud_matt ah, true. This however means that the copy needs to go through a local client (download & upload) and apparently also reencrypts data (with the potential of deduplication not working).

https://restic.readthedocs.io/en/latest/045_working_with_repos.html#copying-snapshots-between-repositories

Its been ages since I last looked into restic, as it "just worked" for me. The thing to highlight is that apparently there has been a successful handover in maintainership in the past, as the original author hasn't done any work himself in quite a while.

Another upside is its portability due to being written in golang.

Edit: if one wants to look into restic, https://autorestic.vercel.app/ is a nice wrapper to simplify setup and handling. My old systems still use bits of bash for that.

[MooCloud_Matt]

@fbartels said in [Backups] Ability to add multiple storage provider/location:

local client (download & upload)

True but is Go, and a Storage Server/VPS have enough resources to have both server and client on it to replicate to an cold s3 storage.

@fbartels said in [Backups] Ability to add multiple storage provider/location:

Another upside is its portability due to being written in golang.

We also need to take into consideration that setup a golang script is easy and can be done by almost anybody in the cloudron community and tanks to the Rest API, cloudron can manage the server super easily, instead if it will be selected a nodejs or py script it can be a lot more difficult to setup.
I think that docker should be excluded as a tool to distribute the destination server for the backup, because many VPS provider of Storage Server offers OpenVZ and not KVM, we would not have that issue but many the will.

[robi]

Don't forget about rclone.org

[MooCloud_Matt]

@robi
I actually don't like that the 3-2-1 is managed by the main server, because if that is compromised you will have compromised also your backup.

I think that: if cloudron wants to offer a better backup solution should have a 3° party software/node be in charge of the replication for the 2-1.
This will protect the server from any ransomware or if your server is compromised.

[marcusquinn]

Borrowing an answer from StackOverflow that may work:

• Minio Cloudron instance: using the command mc mirror on a cron job.

If that works, it could just be a case of documenting and maybe a GUI to make it user-friendly @girish ?

[MooCloud_Matt]

@marcusquinn
a full install of cloudron is too many resources w8st for many storage servers, we speak of old CPU (many of our storage servers have Haswell xeon) or just 1 vCore (time4vps) , and often without the support of docker.

[robi]

@moocloud_matt said in [Backups] Ability to add multiple storage provider/location:

I actually don't like that the 3-2-1 is managed by the main server, because if that is compromised you will have compromised also your backup.

That's the problem with traditional backups.

Next gen way of thinking about backups is simply having a much more resilient storage system. For example, when your data is sprinkled across 8 places and you only need 5 to restore any file/object. There are some very clever and efficient algorithms for this m of n approach which removes the need for 3x replication.

Minio can do this, and as a community we can pool resources to have 20+ places and only need 7 or so to be available at any one time. Maybe even start a coop.

[neurokrish]

@robi great idea. +1 for m of n approach with minio!