Bare Metal Server Backup/Rescue Software

[joseph]

If you want to backup the OS, you need use disk backup tools. Something like https://clonezilla.org/ ? Another idea is to use a filesystem that supports snapshotting like btrfs.

[crazybrad]

@joseph Thank you. Clonezilla was next on my list to consider, but I will also read about btrfs.

[fbartels]

Hi @crazybrad,

just food for thought. But if data recovery is already separated out, instead of treating the server as a pet, automate its installation and maintenance with e.g. Ansible & Terraform and let it be cattle.

Full disk backups are for sure nice, but generating them requires the server to be offline. Plus if you don't pull regular automated backups it will happen that someone changes a configuration file to fix an issue, and when the server is the restored from backups months later you may be missing manual fixes.

If you still want to go with the full machine backup, https://www.veeam.com/ is a popular solution in this space.

[humptydumpty]

Not what you're looking for, but worth a mention is a tool called Ventoy.

Ventoy is a free and open-source utility used for creating bootable usb media storage device with files such as.iso,.wim,.img,.vhd, and.efi files. 
Once Ventoy is installed onto a USB drive, there is no need to reformat the disk to update it with new installation files; it is enough to copy the.iso,.wim,.img,.vhd, or.efi file to the USB drive and boot from them directly.

https://github.com/ventoy/Ventoy

[fbartels]

@humptydumpty said in Bare Metal Server Backup/Rescue Software:

https://github.com/ventoy/Ventoy

On a similar note: https://netboot.xyz/ is nice as well for easy installations

[crazybrad]

@fbartels Moooo! What an interesting idea. We are using Ansible at a higher level but never considered managing a server config using it (or Terraform). Have you come across an Ansible playbook that you could share as an example of how to do this? You are also making me rethink about how we deploy cloud servers. Currently, they are pets that we manually clone using the same written process. Now I have much more to think about. Thanks!

[crazybrad]

@humptydumpty Really interesting tool. I can see a Swiss Army knife of bootable OSes in my future. Thank you.

[crazybrad]

@fbartels Another interesting tool. And using Ansible as well under the hood. Much appreciated.

[fbartels]

@crazybrad how can you use Ansible for anything else than managing systems?

You can use Terraform for the init of the server. There is a module from Supermicro for it. https://github.com/supermicro/redfish/blob/ea5ea99eda0b7ac50d48ca7faa307dfcd3f41e05/Postman_Collections/05_managers.postman_collection.json#L3025-L3074

And then you finish up the installation with Ansible. Cloud-init or the good old preseed for Debian/Ubuntu could be things to look into as well.

A different concept to look into are immutable servers.

[crazybrad]

@fbartels Looks like I have a few months of "no weekends" ahead of me to explore this. Really powerful stuff. NetBoot.xyz looks really interesting. I have never implemented PXE boot, but this seems like a great way to store snapshots and make them accessible. @humptydumpty Ventoy is a winner! I was able to add multiple images (CloneZilla, Ubuntu 24.04 Server, Ubuntu 24.04 Desktop). This has saved a ton of time during my tests. Now I am off to tackle a SuperMicro server with 2 USB ports and 3 USB devices required. Anker USB Hub to the rescue (I hope).

Many thanks to all for helping me get to this point.

[crazybrad]

Final solution: Ventoy + Clonezilla + Anker USB Hub = Tested server restore from USB image!

Future solution(s): Terraform (cattle ranching sounds like fun), NetBoot.xyz, Clonezilla with restore images stored on Amazon S3.

Thanks to everyone for helping me solve this one!