Option to automatically reboot Ubuntu for security updates

[Lonkle]

This is needed nearly every week and it would be so much more helpful if I could just automatically set Cloudron to reboot the server every Sunday at 3AM.

And, my VPS is kind of slow to get back up and running so if I could disable the "app is down" detectors for a little while after reboot so I don't get 5 "your app is down" notifications per reboot, that would be a cool addition as well.

[nebulon]

Ideally we soon can enable ubuntu livepatch by default, then this should go away. Until that it looks like we have some blocking process or so during reboot. We have gotten various reports now that reboots are slow and I also saw that on one of our company Cloudrons yesterday. Unfortunately we are not sure what happens yet.

[Lonkle]

@nebulon I was wondering about the reboot process. If I reboot from my main VPS control panel. Rebooting is fairly quick (a few minutes). But if I reboot with Cloudron, I have to wait 10 - 15 minutes till everything is back up. I thought maybe it was because it was applying security updates, but it happens even when I reboot the VPS from within Cloudron just even without security updates to install.

Ubuntu Live patches would solve the problem for sure so that's exciting. The only annoying part of the slow reboot is that I get so many "app is down" notifications by the time it's fully rebooted. Which means Cloudron starts running its checks before it's fully rebooted. Neither issue exists when opening from the VPS control panel. I was always curious why but I'm glad to know I'm not the only one and even more glad to know that Ubuntu Livepatch support is coming. Do you think that will be ready for 6.0?

[d19dotca]

@Lonk This is partly discussed (with reboots not working well) in this thread here: https://forum.cloudron.io/topic/3130/ubuntu-reboot-to-install-updates-left-some-apps-not-responding?_=1601066258800 - just linking for completeness.

[Lonkle]

Thanks for sharing that @d19dotca. Personally, my apps eventually all come on, but not before a few “down” notifications and 10 - 15 minutes. If I reboot the OS from within the VPS control panel. There’s never an issue. So I thought it was related to the security updates themselves or maybe the reboot function within the Cloudon app is doing something it shouldn’t. Or not doing something it should.

[mehdi]

Isn't it simply that the server takes a long time to shut down, but a normal time to boot up ? I guess it would explain all these symptoms. When rebooting from the VPS control panel, it's a hard shutdown, so there's no time lost.

I guess someone who has a server at home could test this hypothesis.

[nebulon]

We removed the explicit sync in the reboot script now: https://git.cloudron.io/cloudron/box/-/commit/dd75cdb37ed751b31c35755e5e8c2f96daeec81b

Lets see if this fixes the slow reboots at least. As mentioned in that commit, running sync on a system which is busy using the disks, it can take a long time. So usually first the processes have to be terminated and then the sync should be issued. This is what happens from now on then. I don't even remember why we put that sync there in the first place.

[Lonkle]

@mehdi You know what, I probably thought the virtual VPS screen was showing me a boot up animation instead it what it was actually showing me, the shut down animation. TIL I have no idea what the current Ubuntu startup animation looks like.

Also, I've wanted to ask ya, @nebulon, I know the Docker base image is based on 20.04 (Bionic?), but the Cloudron platform is built on 18.04. Is there a plan to update that one day or do you feel as long as Ubuntu supports it with security updates, there are no benefits to update?

[nebulon]

The base image is on Bionic Beaver which is actually 18.04 LTS http://releases.ubuntu.com/18.04/
There is no technical reason to have the base image be the same Ubuntu version or even the same Linux distribution as the host system.

Ubuntu 20.04 support will come soon for the host system, but the base image will remain on 18.04 for some time, since changing that requires retesting and fixing all app packages for no real reason at the moment.

The most important aspect is that the versions are still supported for security updates.

[Lonkle]

@nebulon So it was completely the other way around. You're updating the host version (what Cloudron runs on) to 20.04 (Focal), but leaving the DOCKERIMAGE FROM base version at 18.04 (Bionic) until at least security patch support stops coming out for it. Did I get that correct this time?

Sorry, new to the Ubuntu world.

[imc67]

@nebulon and @girish somehow this change (or something else) made the reboot of my 3 cloudron servers superfast!

Before it took at least 4-6 minutes and today everything (all services and apps) are up again within 1 minute!!!!

Good job!!

[d19dotca]

@imc67 Same here. Was really fast tonight for security updates.

[DanTheMan]

Mine was really noticable faster too. Running Ubuntu 18.04 OS here.
Reboot and apps up and running in less than a minute. Like 40 to 45 seconds in total.

Keep up the good work Cloudron staff

[jdaviescoates]

Wow. 🤯

It's WAY faster now.

Like, lightning fast.

I just did a reboot to finish a Ubuntu security update and everything was back up and running in less than 60 seconds.

I almost couldn't believe it!

Thanks!

[Lonkle]

Same here, less than a minute now. Still looking forward to live patches. But I reboot a lot to change the way apphealthmonitor.js works (making it more accurate) and I need to reboot after every change so this is just a godsend.

[d19dotca]

Guess that “sync” was the problem then, though weird if it’s been in there for years already without issues. Maybe something just incompatible after certain updates were applied which caused sync to take so long.

[robi]

@DanTheMan
Likely due to improvements in 18.04.4 and above.

@Lonk
To avoid having to reboot the box, you can run your tests in a docker container using the sysbox-runc for full OS (machine image) capability in a docker container.

Then just reboot the sysbox container.
See requirements: https://github.com/nestybox/sysbox-ee/blob/master/docs/distro-compat.md

[Lonkle]

@robi Interesting feature. Could save development time. Thanks!

[warg]

Is there any news on this? What shall we do if the only sysadmin is on vacation/sick for 4 weeks? A automatic reboot in case it's needed for security updates would be good.

[nebulon]

I guess it may make sense to add an optional automatic reboot in such cases, although this may cause downtime in unexpected timeslots if apps are used in multiple timezones, especially if reboot takes longer.