Fix for kernel bug in Ubuntu 20.04 causing various issues

[girish]

@jdaviescoates Updating to ubuntu 22.04 is a much riskier endeavor than just downgrading the kernel. Downgrading kernel only takes 5 mins. In fact, just this weekend I upgraded all our servers from Ubuntu 18 to 20 and all of them just completely hosed Each one failed in different places - one in upgrading cloud-init, another is still stuck in some "conflicting package" and for another I had to switch from DO mirror to canonical's mirror. I have generally not had good experiences with distro upgrades (on the server atleast). On desktop ubuntu, I feel things are better, maybe because I have the PC in front of me and have more control of the boot loader.

Ubuntu 22 should work fine though with Cloudron 7.3. But note that it requires you to also rebuild all containers because of cgroup v1 to cgroup v2 migration. All this is in the docs but atleast we had 2-3 bug reports of the migration script not working perfectly.

[girish]

@p44 said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

Should we apply fix to all instances, even those that seemingly don't show any problems?

I think it's best to apply to all instances running that kernel.

What will be next? When they fix kernel issue, we have to operate again to remove fix that we did?

Yes, I think we can unhold the kernel packages and then it will keep auto updating to latest kernel.

[jdaviescoates]

@girish thanks, makes sense.

I guess rather than updating to 22.04 (if people wanted to try that route) it could be safer/ easier to migrate to a fresh install of 22.04?

@imc67 fyi I just did the fix above on my primary netcup server (the other one is only running an unloved instance of Uptime Kuma atm), very easy and seems to have gone smoothly.

[jdaviescoates]

@jdaviescoates said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

seems to have gone smoothly.

Not so fast, some how my backup mount lost its permission and now I'm unable to remount it

[girish]

@jdaviescoates said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

I guess rather than updating to 22.04 (if people wanted to try that route) it could be safer/ easier to migrate to a fresh install of 22.04?

yes, definitely.

[p44]

@girish Hello Girish,

after fix «automatic updates appear to get stuck in 'cleaning up old install"» has been solved, but it seems "cron jobs don't work anymore" problem is still there...

Do you have any other feedback on this issue?

[jdaviescoates]

@p44 said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

"cron jobs don't work anymore" problem is still there.

Seems to have gone for me. Previously my Nextclouds were giving me warnings about that, but they aren't doing that anymore.

[p44]

@jdaviescoates Thank's a lot, I'll do more accurate tests ... it seems only few cron jobs are executed, in external cron panel

[privsec]

Is this a need to be done?

Can this get automated by Cloudron?

[girish]

@privsec It's not automated. I think automating kernel updates would be quite a reach for us since we don't control or have access to the hardware. We have to rely on ubuntu/debian to get their testing right here.

[avatar1024]

@girish said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

I upgraded all our servers from Ubuntu 18 to 20 and all of them just completely hosed

Interesting.... I upgraded one server from 16 > 18 > 20 and two more servers from 18 > 20 using the Cloudron guides and never had a problem. I was going to upgrade to 22 thinking it'll all be easy but a little more unsure now.

[girish]

@avatar1024 I think it's some issue with the DO apt mirrors. Something is out of rsync. I test out upgrading in vultr/linode and they seem perfect.

[girish]

Ubuntu released a new kenel with the fix 5.4.0-135.152 - https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1996678/comments/28 . I don't know if this kernel arrives as a security update.

[nebulon]

At least on vultr Ubuntu 20.04 repository mirrors, the new fixed kernel is already available via security updates. You can check if this is the same in your instance by running:

apt-get update && apt list --upgradable | grep "\-security"

If it lists linux-generic/focal-updates,focal-security 5.4.0.135.133 amd64 [upgradable from: 5.4.0.132.132] then you have to unhold the previously hold packages and eventually it will normally update:

apt-mark unhold linux-generic linux-image-generic linux-headers-generic

[p44]

@girish Hello Girish, did you check if this kernel arrives as a security update?

Can we unhold previously hold packages?

Thank's a lot

[girish]

@p44 this seems to have come to our servers atleast. So, yes, go ahead an unmark the holds.

Start-Date: 2022-12-03  06:10:57
Commandline: /usr/bin/unattended-upgrade
Install: linux-image-5.4.0-135-generic:amd64 (5.4.0-135.152, automatic), linux-headers-5.4.0-135:amd64 (5.4.0-135.152, automatic), linux-modules-extra-5.4.0-135-generic:amd64 (5.4.0-135.152, automatic), linux-modules-5.4.0-135-generic:amd64 (5.4.0-135.152, automatic), linux-headers-5.4.0-135-generic:amd64 (5.4.0-135.152, automatic)
Upgrade: linux-headers-generic:amd64 (5.4.0.132.132, 5.4.0.135.133), linux-image-generic:amd64 (5.4.0.132.132, 5.4.0.135.133), linux-generic:amd64 (5.4.0.132.132, 5.4.0.135.133)
End-Date: 2022-12-03  06:12:14

[p44]

@girish Ok, thank's a lot

[p44]

@nebulon Should we have to reverse also this step?

• Edit /etc/default/grub . Find the line GRUB_DEFAULT=0. Change this to GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 5.4.0-131-generic" . Important you get this line right, otherwise your server may not boot

[girish]

@p44 yes

[p44]

@girish @nebulon please can you post here all the whole process to reverse this fix?