Immutable server os

[fbartels]

This post was slightly triggered by https://forum.cloudron.io/post/88508, which brings up the point that while Cloudron does a lot of work for the admin of the system, the one thing it does not do is upgrade the OS from one release to the next. And to be fair, OS upgrades are/can be difficult, and I am not entirely sure I would want my server to do a Ubuntu dist upgrade all by itself.

The last few weeks I have been playing around with immutable operating systems. My own goal was to see how I could reduce the maintenance burden for applications not running on my Cloudron. Immutable systems have a mostly read-only system with dedicated locations for write access (hey, that sounds like how applications are run on Cloudron) and typically use some form of containerisation (docker/oci, snap, flatpak) to install any runtime applications, which is what Cloudron does for applications and addons, but there are also components on Cloudron that are installed directly on the host, like nginx for example.

Ubuntu has its "Core" variant (see also) and while I am usually a fan of Debian-based systems, the most promising system for me so far is Fedora CoreOS (fcos). Since fcos is often used as a base for Kubernetes clusters, there is already a lot of documentation and installation options for hosting providers. An interesting perspective on fcos can be found at https://blog.gripdev.xyz/2024/03/16/in-search-of-a-zero-toil-homelab-with-immutable-linux/, but in the end the system can be summarised as follows:

• You boot the installation media and either automatically or manually provide it with an "ignition" file.
  • The ignition file is JSON and is generated by Butane from a YAML document.
  • This file contains the deployment information for the system, such as users, passwords/SSH keys, files to be added to the final system, partition layout and file systems (if you want to customise them), ...
• once installed, the server will periodically check for updates and simply reboot to a newer version at a defined interval (with the ability to revert to the previous state if something goes wrong)
  • In a cluster, there is a service that coordinates these reboots, but in the simplest case, you can create a file locally so that Zincati just does this reboot e.g. every Monday night between 22:00 and 22:30.
• fcos comes with the docker/moby daemon natively, but being a Red Hat system, it also comes with all the bits of systemd and podman.
• and one interesting concept in fcos that I haven't yet toyed with is the ability instead boot into an own container image, which can be customised to include whatever tools and dependencies you need to stand up your environment.

Have others also made some experiences with immutable servers? What has been your choice of platform so far?

I think if if am in the mood to (shortly) sink my productivity I am going to switch my desktop system over to VanillaOS some time in the near future.

[timconsidine]

@fbartels very interesting, as I was not aware of immutable os

[girish]

Thanks for the great links! I wasn't aware of Ubuntu Core even. There goes my weekend to investigate...

I had great hopes for CoreOS which was trying to run containers "natively" . I thought redhat might take it further, but unfortunately, that project died. I was really hoping to use CoreOS for Cloudron. Would also love to see a "standardized" packaging mechanism for selfhosted apps on servers. Unfortunately, all the talk is now about k8s and people get distracted and write overly complicated specs (like the helm stuff).

[fbartels]

@girish said in Immutable server os:

There goes my weekend to investigate...

I am sorry for that but to be honest I have also spent good amounts of past weekends diving into this topic.

@girish said in Immutable server os:

I had great hopes for CoreOS

Yes, fcos is actually the successor of this, but has meanwhile gotten a bunch of new components such as the usage of ignition instead of cloud-init.

I forgot to mention one other cool this about fcos. Once you have generated your ignition file you can use the coreos-installer to bake it into an iso file. I don't know if there is an easier way to generate a software appliance out there. And inside of the ignition file you can merge or even completely replace it at runtime with files it can fetch from the internet at runtime.

https://major.io/p/coreos-as-pet/ is another blog that highlights some of the potentials of fcos outside of the usual Kubernetes usage.