Bare Metal Server Backup/Rescue Software

[crazybrad]

Couldn't think of a better group of tech experts than "Cloudronites" to pose this question to. (The use case is a little unusual as the bare metal server would not be used to host a Cloudron server per se, but the server will be used to provide specialized, custom solutions to Cloudron users).

I need to deploy a disaster recovery solution to a SuperMicro server (multiprocessor w/ hardware RAID1) located in a remote data center running Ubuntu 24.04. I need to be able to restore roughly 20GB of OS and installed software in the event of a corrupted server without skilled engineers to troubleshoot this. Data recovery is not a concern as that is being handled by a separate process. In addition, it is not necessary to constantly refresh the rescue media as Ubuntu APT can easily update the restored server from the older rescue snapshot. [Note: For those thinking "why not go cloud instead, there are special cost reasons in this case to go bare metal and endure the hassle.]

The ideal solution would be simple, open source, and would allow for creation of a rescue USB that could be used with the help of remote "hands" to boot and restore the server and replace corrupted partitions automatically. If the solution created periodic snapshots (nice, but not required), storage at Amazon S3 or equivalent would be ideal.

We have looked at REAR (Relax and Recover), Bacula, and Borg with no clear answers as to what might be best. We are also open to other solutions as well.

Thanks in advance for any and all opinions.

[joseph]

If you want to backup the OS, you need use disk backup tools. Something like https://clonezilla.org/ ? Another idea is to use a filesystem that supports snapshotting like btrfs.

[crazybrad]

@joseph Thank you. Clonezilla was next on my list to consider, but I will also read about btrfs.

[fbartels]

Hi @crazybrad,

just food for thought. But if data recovery is already separated out, instead of treating the server as a pet, automate its installation and maintenance with e.g. Ansible & Terraform and let it be cattle.

Full disk backups are for sure nice, but generating them requires the server to be offline. Plus if you don't pull regular automated backups it will happen that someone changes a configuration file to fix an issue, and when the server is the restored from backups months later you may be missing manual fixes.

If you still want to go with the full machine backup, https://www.veeam.com/ is a popular solution in this space.

[humptydumpty]

Not what you're looking for, but worth a mention is a tool called Ventoy.

Ventoy is a free and open-source utility used for creating bootable usb media storage device with files such as.iso,.wim,.img,.vhd, and.efi files. 
Once Ventoy is installed onto a USB drive, there is no need to reformat the disk to update it with new installation files; it is enough to copy the.iso,.wim,.img,.vhd, or.efi file to the USB drive and boot from them directly.

https://github.com/ventoy/Ventoy

[fbartels]

@humptydumpty said in Bare Metal Server Backup/Rescue Software:

https://github.com/ventoy/Ventoy

On a similar note: https://netboot.xyz/ is nice as well for easy installations

[crazybrad]

@fbartels Moooo! What an interesting idea. We are using Ansible at a higher level but never considered managing a server config using it (or Terraform). Have you come across an Ansible playbook that you could share as an example of how to do this? You are also making me rethink about how we deploy cloud servers. Currently, they are pets that we manually clone using the same written process. Now I have much more to think about. Thanks!

[crazybrad]

@humptydumpty Really interesting tool. I can see a Swiss Army knife of bootable OSes in my future. Thank you.

[crazybrad]

@fbartels Another interesting tool. And using Ansible as well under the hood. Much appreciated.

[fbartels]

@crazybrad how can you use Ansible for anything else than managing systems?

You can use Terraform for the init of the server. There is a module from Supermicro for it. https://github.com/supermicro/redfish/blob/ea5ea99eda0b7ac50d48ca7faa307dfcd3f41e05/Postman_Collections/05_managers.postman_collection.json#L3025-L3074

And then you finish up the installation with Ansible. Cloud-init or the good old preseed for Debian/Ubuntu could be things to look into as well.

A different concept to look into are immutable servers.

[crazybrad]

@fbartels Looks like I have a few months of "no weekends" ahead of me to explore this. Really powerful stuff. NetBoot.xyz looks really interesting. I have never implemented PXE boot, but this seems like a great way to store snapshots and make them accessible. @humptydumpty Ventoy is a winner! I was able to add multiple images (CloneZilla, Ubuntu 24.04 Server, Ubuntu 24.04 Desktop). This has saved a ton of time during my tests. Now I am off to tackle a SuperMicro server with 2 USB ports and 3 USB devices required. Anker USB Hub to the rescue (I hope).

Many thanks to all for helping me get to this point.

[crazybrad]

Final solution: Ventoy + Clonezilla + Anker USB Hub = Tested server restore from USB image!

Future solution(s): Terraform (cattle ranching sounds like fun), NetBoot.xyz, Clonezilla with restore images stored on Amazon S3.

Thanks to everyone for helping me solve this one!