Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse

Cloudron Forum

Apps | Demo | Docs | Install

Fix for kernel bug in Ubuntu 20.04 causing various issues

Scheduled Pinned Locked Moved Solved Support
44 Posts 8 Posters 2.8k Views
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • girishG Do not disturb
    girishG Do not disturb
    girish Staff
    wrote on last edited by girish
    #1

    This fix is not needed anymore. Ubuntu has released 5.4.0-135-generic

    Cloudron enables automatic ubuntu security updates. Roughly around 2022-11-17, the linux kernel was updated to 5.4.0-132-generic . You can find the automatic updates log in /var/log/apt/history.log. This kernel has a bug causing various things like containerd, prometheus node exporter etc to fail. On Cloudron, this manifests itself as:

    • automatic updates appear to get stuck in 'cleaning up old install"
    • cron jobs don't work anymore
    • file permissions inside containers become incorrect

    For the moment, it's best to revert to the previous kernel 5.4.0-131-generic. How you do this, depends on your VPS provider. Some VPS providers allow you to change the kernel via their control panels.

    Please be careful with instructions below. You might have to fine tune it based on your setup/provider.

    Many of the modern provider will just use Grub 2 as the kernel (digitalocean, linode, to name a few). On such VPS, please change the kernel as follows:

    • Highly recommend taking a snapshot of the server, in case something goes wrong.
    • SSH into the server
    • apt install linux-image-5.4.0-131-generic linux-modules-extra-5.4.0-131-generic
    • apt-mark hold linux-generic linux-image-generic linux-headers-generic
    • Edit /etc/default/grub . Find the line GRUB_DEFAULT=0. Change this to GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 5.4.0-131-generic" . Important you get this line right, otherwise your server may not boot!
    • update-grub
    • reboot
    • After reboot, uname -nar will say 5.4.0-131-generic.

    To reverse the above changes:

    • apt-mark unhold linux-generic linux-image-generic linux-headers-generic
    • unattended-upgrade -d - when running this you will see new kernel 5.4.0-135-generic is getting installed.
    • Edit /etc/default/grub . Change the line to GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 5.4.0-135-generic".
    • update-grub
    • reboot
    • After reboot, uname -nar will say 5.4.0-135-generic
    • apt remove linux-image-5.4.0-131-generic linux-modules-extra-5.4.0-131-generic - to remove the old kernel
    • Edit /etc/default/grub . Change the line to `GRUB_DEFAULT=0
    • update-grub

    Related threads:

    • https://forum.cloudron.io/topic/8096/docker-needs-to-be-restarted-often-due-to-fail-app-updates
    • https://forum.cloudron.io/topic/8032/apps-stuck-updating-cleaning-up-old-install-even-after-stopping-restarting-task
    imc67I P O 3 Replies Last reply
    10
  • girishG girish pinned this topic on
  • girishG girish referenced this topic on
  • girishG girish referenced this topic on
  • imc67I Offline
    imc67I Offline
    imc67 translator
    replied to girish on last edited by
    #2

    @girish was anyone succesful in reverting the kernel (especially on NetCup)?

    jdaviescoatesJ 1 Reply Last reply
    1
  • jdaviescoatesJ Offline
    jdaviescoatesJ Offline
    jdaviescoates
    replied to imc67 on last edited by
    #3

    @imc67 I've not tried yet, but I'm interested in why @girish recommends the kernel reversion as opposed to updating Ubuntu 22.04?

    I use Cloudron with Gandi & Hetzner

    imc67I 1 Reply Last reply
    2
  • imc67I Offline
    imc67I Offline
    imc67 translator
    replied to jdaviescoates on last edited by imc67
    #4

    @jdaviescoates said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

    @imc67 I've not tried yet, but I'm interested in why @girish recommends the kernel reversion as opposed to updating Ubuntu 22.04?

    was thinking about that too, but then first you need to update Cloudron to 7.3.2 and that's not stable yet? Or won't even update because of the issues?

    jdaviescoatesJ 1 Reply Last reply
    0
  • jdaviescoatesJ Offline
    jdaviescoatesJ Offline
    jdaviescoates
    replied to imc67 on last edited by
    #5

    @imc67 said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

    update Cloudron to 7.3.2 and that's not stable yet?

    I'm already on it.

    @imc67 said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

    Or won't even update because of the issues?

    That could be why I guess, might be risky to update with the issue.

    But still wondering if it might be the best option for me given I'm already on 7.3.2

    I use Cloudron with Gandi & Hetzner

    girishG 1 Reply Last reply
    1
  • P Offline
    P Offline
    p44 translator
    replied to girish on last edited by p44
    #6

    @girish Ok thank's, I applied this https://forum.cloudron.io/topic/8101/fix-for-kernel-bug-in-ubuntu-20-04-causing-various-issues on most of active instances.

    • Should we apply fix to all instances, even those that seemingly don't show any problems?
    • What will be next? When they fix kernel issue, we have to operate again to remove fix that we did?

    Thank's a lot

    girishG 1 Reply Last reply
    1
  • P p44 referenced this topic on
  • girishG Do not disturb
    girishG Do not disturb
    girish Staff
    replied to jdaviescoates on last edited by
    #7

    @jdaviescoates Updating to ubuntu 22.04 is a much riskier endeavor than just downgrading the kernel. Downgrading kernel only takes 5 mins. In fact, just this weekend I upgraded all our servers from Ubuntu 18 to 20 and all of them just completely hosed 😞 Each one failed in different places - one in upgrading cloud-init, another is still stuck in some "conflicting package" and for another I had to switch from DO mirror to canonical's mirror. I have generally not had good experiences with distro upgrades (on the server atleast). On desktop ubuntu, I feel things are better, maybe because I have the PC in front of me and have more control of the boot loader.

    Ubuntu 22 should work fine though with Cloudron 7.3. But note that it requires you to also rebuild all containers because of cgroup v1 to cgroup v2 migration. All this is in the docs but atleast we had 2-3 bug reports of the migration script not working perfectly.

    avatar1024A 1 Reply Last reply
    2
  • girishG Do not disturb
    girishG Do not disturb
    girish Staff
    replied to p44 on last edited by
    #8

    @p44 said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

    Should we apply fix to all instances, even those that seemingly don't show any problems?

    I think it's best to apply to all instances running that kernel.

    What will be next? When they fix kernel issue, we have to operate again to remove fix that we did?

    Yes, I think we can unhold the kernel packages and then it will keep auto updating to latest kernel.

    jdaviescoatesJ 1 Reply Last reply
    2
  • jdaviescoatesJ Offline
    jdaviescoatesJ Offline
    jdaviescoates
    replied to girish on last edited by jdaviescoates
    #9

    @girish thanks, makes sense.

    I guess rather than updating to 22.04 (if people wanted to try that route) it could be safer/ easier to migrate to a fresh install of 22.04?

    @imc67 fyi I just did the fix above on my primary netcup server (the other one is only running an unloved instance of Uptime Kuma atm), very easy and seems to have gone smoothly.

    I use Cloudron with Gandi & Hetzner

    jdaviescoatesJ girishG 2 Replies Last reply
    4
  • jdaviescoatesJ Offline
    jdaviescoatesJ Offline
    jdaviescoates
    replied to jdaviescoates on last edited by
    #10

    @jdaviescoates said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

    seems to have gone smoothly.

    Not so fast, some how my backup mount lost its permission and now I'm unable to remount it

    I use Cloudron with Gandi & Hetzner

    1 Reply Last reply
    1
  • girishG Do not disturb
    girishG Do not disturb
    girish Staff
    replied to jdaviescoates on last edited by
    #11

    @jdaviescoates said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

    I guess rather than updating to 22.04 (if people wanted to try that route) it could be safer/ easier to migrate to a fresh install of 22.04?

    yes, definitely.

    martinkbsM 1 Reply Last reply
    1
  • girishG girish forked this topic on
  • jdaviescoatesJ jdaviescoates referenced this topic on
  • P Offline
    P Offline
    p44 translator
    replied to girish on last edited by p44
    #12

    @girish Hello Girish,

    after fix «automatic updates appear to get stuck in 'cleaning up old install"» has been solved, but it seems "cron jobs don't work anymore" problem is still there...

    Do you have any other feedback on this issue?

    jdaviescoatesJ 1 Reply Last reply
    0
  • jdaviescoatesJ Offline
    jdaviescoatesJ Offline
    jdaviescoates
    replied to p44 on last edited by
    #13

    @p44 said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

    "cron jobs don't work anymore" problem is still there.

    Seems to have gone for me. Previously my Nextclouds were giving me warnings about that, but they aren't doing that anymore.

    I use Cloudron with Gandi & Hetzner

    P 1 Reply Last reply
    2
  • P Offline
    P Offline
    p44 translator
    replied to jdaviescoates on last edited by
    #14

    @jdaviescoates Thank's a lot, I'll do more accurate tests ... it seems only few cron jobs are executed, in external cron panel

    1 Reply Last reply
    0
  • P Offline
    P Offline
    privsec
    wrote on last edited by
    #15

    Is this a need to be done?

    Can this get automated by Cloudron?

    girishG 1 Reply Last reply
    0
  • girishG Do not disturb
    girishG Do not disturb
    girish Staff
    replied to privsec on last edited by
    #16

    @privsec It's not automated. I think automating kernel updates would be quite a reach for us since we don't control or have access to the hardware. We have to rely on ubuntu/debian to get their testing right here.

    1 Reply Last reply
    3
  • avatar1024A Offline
    avatar1024A Offline
    avatar1024
    replied to girish on last edited by
    #17

    @girish said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

    I upgraded all our servers from Ubuntu 18 to 20 and all of them just completely hosed

    Interesting.... I upgraded one server from 16 > 18 > 20 and two more servers from 18 > 20 using the Cloudron guides and never had a problem. I was going to upgrade to 22 thinking it'll all be easy but a little more unsure now.

    girishG 1 Reply Last reply
    1
  • girishG Do not disturb
    girishG Do not disturb
    girish Staff
    replied to avatar1024 on last edited by
    #18

    @avatar1024 I think it's some issue with the DO apt mirrors. Something is out of rsync. I test out upgrading in vultr/linode and they seem perfect.

    1 Reply Last reply
    1
  • BrutalBirdieB BrutalBirdie referenced this topic on
  • girishG Do not disturb
    girishG Do not disturb
    girish Staff
    wrote on last edited by
    #19

    Ubuntu released a new kenel with the fix 5.4.0-135.152 - https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1996678/comments/28 . I don't know if this kernel arrives as a security update.

    P 1 Reply Last reply
    3
  • nebulonN Online
    nebulonN Online
    nebulon Staff
    wrote on last edited by
    #20

    At least on vultr Ubuntu 20.04 repository mirrors, the new fixed kernel is already available via security updates. You can check if this is the same in your instance by running:

    apt-get update && apt list --upgradable | grep "\-security"
    

    If it lists linux-generic/focal-updates,focal-security 5.4.0.135.133 amd64 [upgradable from: 5.4.0.132.132] then you have to unhold the previously hold packages and eventually it will normally update:

    apt-mark unhold linux-generic linux-image-generic linux-headers-generic
    
    P 1 Reply Last reply
    4

  • Login

  • Don't have an account? Register

  • Login or register to search.
  • First post
    Last post
0
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Login

  • Don't have an account? Register

  • Login or register to search.