Fix for kernel bug in Ubuntu 20.04 causing various issues

[p44]

@girish Hello Girish,

after fix «automatic updates appear to get stuck in 'cleaning up old install"» has been solved, but it seems "cron jobs don't work anymore" problem is still there...

Do you have any other feedback on this issue?

[jdaviescoates]

@p44 said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

"cron jobs don't work anymore" problem is still there.

Seems to have gone for me. Previously my Nextclouds were giving me warnings about that, but they aren't doing that anymore.

[p44]

@jdaviescoates Thank's a lot, I'll do more accurate tests ... it seems only few cron jobs are executed, in external cron panel

[privsec]

Is this a need to be done?

Can this get automated by Cloudron?

[girish]

@privsec It's not automated. I think automating kernel updates would be quite a reach for us since we don't control or have access to the hardware. We have to rely on ubuntu/debian to get their testing right here.

[avatar1024]

@girish said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

I upgraded all our servers from Ubuntu 18 to 20 and all of them just completely hosed

Interesting.... I upgraded one server from 16 > 18 > 20 and two more servers from 18 > 20 using the Cloudron guides and never had a problem. I was going to upgrade to 22 thinking it'll all be easy but a little more unsure now.

[girish]

@avatar1024 I think it's some issue with the DO apt mirrors. Something is out of rsync. I test out upgrading in vultr/linode and they seem perfect.

[girish]

Ubuntu released a new kenel with the fix 5.4.0-135.152 - https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1996678/comments/28 . I don't know if this kernel arrives as a security update.

[nebulon]

At least on vultr Ubuntu 20.04 repository mirrors, the new fixed kernel is already available via security updates. You can check if this is the same in your instance by running:

apt-get update && apt list --upgradable | grep "\-security"

If it lists linux-generic/focal-updates,focal-security 5.4.0.135.133 amd64 [upgradable from: 5.4.0.132.132] then you have to unhold the previously hold packages and eventually it will normally update:

apt-mark unhold linux-generic linux-image-generic linux-headers-generic

[p44]

@girish Hello Girish, did you check if this kernel arrives as a security update?

Can we unhold previously hold packages?

Thank's a lot

[girish]

@p44 this seems to have come to our servers atleast. So, yes, go ahead an unmark the holds.

Start-Date: 2022-12-03  06:10:57
Commandline: /usr/bin/unattended-upgrade
Install: linux-image-5.4.0-135-generic:amd64 (5.4.0-135.152, automatic), linux-headers-5.4.0-135:amd64 (5.4.0-135.152, automatic), linux-modules-extra-5.4.0-135-generic:amd64 (5.4.0-135.152, automatic), linux-modules-5.4.0-135-generic:amd64 (5.4.0-135.152, automatic), linux-headers-5.4.0-135-generic:amd64 (5.4.0-135.152, automatic)
Upgrade: linux-headers-generic:amd64 (5.4.0.132.132, 5.4.0.135.133), linux-image-generic:amd64 (5.4.0.132.132, 5.4.0.135.133), linux-generic:amd64 (5.4.0.132.132, 5.4.0.135.133)
End-Date: 2022-12-03  06:12:14

[p44]

@girish Ok, thank's a lot

[p44]

@nebulon Should we have to reverse also this step?

• Edit /etc/default/grub . Find the line GRUB_DEFAULT=0. Change this to GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 5.4.0-131-generic" . Important you get this line right, otherwise your server may not boot

[girish]

@p44 yes

[p44]

@girish @nebulon please can you post here all the whole process to reverse this fix?

[privsec]

@p44 said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

@girish @nebulon please can you post here all the whole process to reverse this fix?

Yes please, the whole process.

[p44]

Hello @girish and @nebulon, when you have time can you take a look to this? Thank's

[girish]

@p44 I put in instructions to reverse in the original post.

[p44]

@girish Thank's a lot

[odie]

@girish said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

@p44 I put in instructions to reverse in the original post.

Thanks.

I tried to implement the instructions to reverse. For me, this didn't work. I followed the steps, did the reboot, and uname -nar still shows 5.4.0-131-generic #147-Ubuntu SMP Fri Oct 14 17:07:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

I tried to follow the steps suggested by @nebulon here: https://forum.cloudron.io/post/57470

When checking for updates, it provided the following output:

Hit:1 http://no.archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://no.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:3 http://no.archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:4 http://no.archive.ubuntu.com/ubuntu focal-security InRelease
Reading package lists... Done

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Maybe it was wrong, but I unholded the previously hold packages.

Then I rebooted and waited a few hours. Nothing happened, I was still on 5.4.0-131-generic

So I tried the next step I could find, by following the instructions here (as best as I could understand): https://forum.cloudron.io/post/57552

At command line, I typed sudo /usr/bin/unattended-upgrade

Again, I rebooted and again I was still on 5.4.0-131 when I checked today.

So now, I tried to follow the new instructions you put on top of this post.

First, the unhold command gave the output that the change already had been unholded.

Second, when doing sudo unattended-upgrade -d, I got a huge output, but nothing in that output said anything about 5.4.0-135 generic. (I pasted the full output below, sorry if it becomes a long post).

I continued to follow the instructions as best as I could, so I edited /etc/default/grub like it says in the reverse instruction post, did a reboot, and checked uname -nar again. Still on 5.4.0-131 generic.

I have not apt removed 5.4.0.131 (as I thought that perhaps would be a bad idea since I apparently don't have a newer), and I have not restored /etc/default/grub to default 0.

Could you offer any assistance, perhaps? Is this easily fixable, or do I have to clean install the whole Cloudron and restore from backup?

I am on Ubuntu 20.04 and I self host.

Thank you!

edit: I have to reply to this post, because the output is too long for the editor: