Option to automatically reboot Ubuntu for security updates

[d19dotca]

@Lonk This is partly discussed (with reboots not working well) in this thread here: https://forum.cloudron.io/topic/3130/ubuntu-reboot-to-install-updates-left-some-apps-not-responding?_=1601066258800 - just linking for completeness.

[Lonkle]

Thanks for sharing that @d19dotca. Personally, my apps eventually all come on, but not before a few “down” notifications and 10 - 15 minutes. If I reboot the OS from within the VPS control panel. There’s never an issue. So I thought it was related to the security updates themselves or maybe the reboot function within the Cloudon app is doing something it shouldn’t. Or not doing something it should.

[mehdi]

Isn't it simply that the server takes a long time to shut down, but a normal time to boot up ? I guess it would explain all these symptoms. When rebooting from the VPS control panel, it's a hard shutdown, so there's no time lost.

I guess someone who has a server at home could test this hypothesis.

[nebulon]

We removed the explicit sync in the reboot script now: https://git.cloudron.io/cloudron/box/-/commit/dd75cdb37ed751b31c35755e5e8c2f96daeec81b

Lets see if this fixes the slow reboots at least. As mentioned in that commit, running sync on a system which is busy using the disks, it can take a long time. So usually first the processes have to be terminated and then the sync should be issued. This is what happens from now on then. I don't even remember why we put that sync there in the first place.

[Lonkle]

@mehdi You know what, I probably thought the virtual VPS screen was showing me a boot up animation instead it what it was actually showing me, the shut down animation. TIL I have no idea what the current Ubuntu startup animation looks like.

Also, I've wanted to ask ya, @nebulon, I know the Docker base image is based on 20.04 (Bionic?), but the Cloudron platform is built on 18.04. Is there a plan to update that one day or do you feel as long as Ubuntu supports it with security updates, there are no benefits to update?

[nebulon]

The base image is on Bionic Beaver which is actually 18.04 LTS http://releases.ubuntu.com/18.04/
There is no technical reason to have the base image be the same Ubuntu version or even the same Linux distribution as the host system.

Ubuntu 20.04 support will come soon for the host system, but the base image will remain on 18.04 for some time, since changing that requires retesting and fixing all app packages for no real reason at the moment.

The most important aspect is that the versions are still supported for security updates.

[Lonkle]

@nebulon So it was completely the other way around. You're updating the host version (what Cloudron runs on) to 20.04 (Focal), but leaving the DOCKERIMAGE FROM base version at 18.04 (Bionic) until at least security patch support stops coming out for it. Did I get that correct this time?

Sorry, new to the Ubuntu world.

[imc67]

@nebulon and @girish somehow this change (or something else) made the reboot of my 3 cloudron servers superfast!

Before it took at least 4-6 minutes and today everything (all services and apps) are up again within 1 minute!!!!

Good job!!

[d19dotca]

@imc67 Same here. Was really fast tonight for security updates.

[DanTheMan]

Mine was really noticable faster too. Running Ubuntu 18.04 OS here.
Reboot and apps up and running in less than a minute. Like 40 to 45 seconds in total.

Keep up the good work Cloudron staff

[jdaviescoates]

Wow. 🤯

It's WAY faster now.

Like, lightning fast.

I just did a reboot to finish a Ubuntu security update and everything was back up and running in less than 60 seconds.

I almost couldn't believe it!

Thanks!

[Lonkle]

Same here, less than a minute now. Still looking forward to live patches. But I reboot a lot to change the way apphealthmonitor.js works (making it more accurate) and I need to reboot after every change so this is just a godsend.

[d19dotca]

Guess that “sync” was the problem then, though weird if it’s been in there for years already without issues. Maybe something just incompatible after certain updates were applied which caused sync to take so long.

[robi]

@DanTheMan
Likely due to improvements in 18.04.4 and above.

@Lonk
To avoid having to reboot the box, you can run your tests in a docker container using the sysbox-runc for full OS (machine image) capability in a docker container.

Then just reboot the sysbox container.
See requirements: https://github.com/nestybox/sysbox-ee/blob/master/docs/distro-compat.md

[Lonkle]

@robi Interesting feature. Could save development time. Thanks!

[warg]

Is there any news on this? What shall we do if the only sysadmin is on vacation/sick for 4 weeks? A automatic reboot in case it's needed for security updates would be good.

[nebulon]

I guess it may make sense to add an optional automatic reboot in such cases, although this may cause downtime in unexpected timeslots if apps are used in multiple timezones, especially if reboot takes longer.

[warg]

Yep. For those cases it shouldn't be too bad: The fixed timeslots/dates can be communicated/planned accordingly and if it's on demand and no admin is available, the users can get told that they should be worried only if downtime is >1h or something like that. That's at least better than no patched server and in worst case coming home and finding a disaster .

[robi]

This can be automated today with ctfreak which can reboot a system if certain conditions are met. Would be good to document such options.

[potemkin_ai]

@nebulon just for information - livepatch doesn't mean you don't have to reboot - it just lets you postpone that moment as convenient.

For sync - a good practice on old good *nix is to issue sync twice before reboot - to make sure the buffers dumped for sure.

On the reboot automation using third-party apps - it's doable; everything is, but why, if you can just add it inside of the system?