Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Support
  3. Fix for kernel bug in Ubuntu 20.04 causing various issues

Fix for kernel bug in Ubuntu 20.04 causing various issues

Scheduled Pinned Locked Moved Solved Support
44 Posts 8 Posters 6.2k Views 9 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • girishG Offline
    girishG Offline
    girish
    Staff
    wrote on last edited by girish
    #1

    This fix is not needed anymore. Ubuntu has released 5.4.0-135-generic

    Cloudron enables automatic ubuntu security updates. Roughly around 2022-11-17, the linux kernel was updated to 5.4.0-132-generic . You can find the automatic updates log in /var/log/apt/history.log. This kernel has a bug causing various things like containerd, prometheus node exporter etc to fail. On Cloudron, this manifests itself as:

    • automatic updates appear to get stuck in 'cleaning up old install"
    • cron jobs don't work anymore
    • file permissions inside containers become incorrect

    For the moment, it's best to revert to the previous kernel 5.4.0-131-generic. How you do this, depends on your VPS provider. Some VPS providers allow you to change the kernel via their control panels.

    Please be careful with instructions below. You might have to fine tune it based on your setup/provider.

    Many of the modern provider will just use Grub 2 as the kernel (digitalocean, linode, to name a few). On such VPS, please change the kernel as follows:

    • Highly recommend taking a snapshot of the server, in case something goes wrong.
    • SSH into the server
    • apt install linux-image-5.4.0-131-generic linux-modules-extra-5.4.0-131-generic
    • apt-mark hold linux-generic linux-image-generic linux-headers-generic
    • Edit /etc/default/grub . Find the line GRUB_DEFAULT=0. Change this to GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 5.4.0-131-generic" . Important you get this line right, otherwise your server may not boot!
    • update-grub
    • reboot
    • After reboot, uname -nar will say 5.4.0-131-generic.

    To reverse the above changes:

    • apt-mark unhold linux-generic linux-image-generic linux-headers-generic
    • unattended-upgrade -d - when running this you will see new kernel 5.4.0-135-generic is getting installed.
    • Edit /etc/default/grub . Change the line to GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 5.4.0-135-generic".
    • update-grub
    • reboot
    • After reboot, uname -nar will say 5.4.0-135-generic
    • apt remove linux-image-5.4.0-131-generic linux-modules-extra-5.4.0-131-generic - to remove the old kernel
    • Edit /etc/default/grub . Change the line to `GRUB_DEFAULT=0
    • update-grub

    Related threads:

    • https://forum.cloudron.io/topic/8096/docker-needs-to-be-restarted-often-due-to-fail-app-updates
    • https://forum.cloudron.io/topic/8032/apps-stuck-updating-cleaning-up-old-install-even-after-stopping-restarting-task
    imc67I P O 3 Replies Last reply
    10
    • girishG girish pinned this topic on
    • girishG girish referenced this topic on
    • girishG girish referenced this topic on
    • girishG girish

      This fix is not needed anymore. Ubuntu has released 5.4.0-135-generic

      Cloudron enables automatic ubuntu security updates. Roughly around 2022-11-17, the linux kernel was updated to 5.4.0-132-generic . You can find the automatic updates log in /var/log/apt/history.log. This kernel has a bug causing various things like containerd, prometheus node exporter etc to fail. On Cloudron, this manifests itself as:

      • automatic updates appear to get stuck in 'cleaning up old install"
      • cron jobs don't work anymore
      • file permissions inside containers become incorrect

      For the moment, it's best to revert to the previous kernel 5.4.0-131-generic. How you do this, depends on your VPS provider. Some VPS providers allow you to change the kernel via their control panels.

      Please be careful with instructions below. You might have to fine tune it based on your setup/provider.

      Many of the modern provider will just use Grub 2 as the kernel (digitalocean, linode, to name a few). On such VPS, please change the kernel as follows:

      • Highly recommend taking a snapshot of the server, in case something goes wrong.
      • SSH into the server
      • apt install linux-image-5.4.0-131-generic linux-modules-extra-5.4.0-131-generic
      • apt-mark hold linux-generic linux-image-generic linux-headers-generic
      • Edit /etc/default/grub . Find the line GRUB_DEFAULT=0. Change this to GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 5.4.0-131-generic" . Important you get this line right, otherwise your server may not boot!
      • update-grub
      • reboot
      • After reboot, uname -nar will say 5.4.0-131-generic.

      To reverse the above changes:

      • apt-mark unhold linux-generic linux-image-generic linux-headers-generic
      • unattended-upgrade -d - when running this you will see new kernel 5.4.0-135-generic is getting installed.
      • Edit /etc/default/grub . Change the line to GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 5.4.0-135-generic".
      • update-grub
      • reboot
      • After reboot, uname -nar will say 5.4.0-135-generic
      • apt remove linux-image-5.4.0-131-generic linux-modules-extra-5.4.0-131-generic - to remove the old kernel
      • Edit /etc/default/grub . Change the line to `GRUB_DEFAULT=0
      • update-grub

      Related threads:

      • https://forum.cloudron.io/topic/8096/docker-needs-to-be-restarted-often-due-to-fail-app-updates
      • https://forum.cloudron.io/topic/8032/apps-stuck-updating-cleaning-up-old-install-even-after-stopping-restarting-task
      imc67I Offline
      imc67I Offline
      imc67
      translator
      wrote on last edited by
      #2

      @girish was anyone succesful in reverting the kernel (especially on NetCup)?

      jdaviescoatesJ 1 Reply Last reply
      1
      • imc67I imc67

        @girish was anyone succesful in reverting the kernel (especially on NetCup)?

        jdaviescoatesJ Offline
        jdaviescoatesJ Offline
        jdaviescoates
        wrote on last edited by
        #3

        @imc67 I've not tried yet, but I'm interested in why @girish recommends the kernel reversion as opposed to updating Ubuntu 22.04?

        I use Cloudron with Gandi & Hetzner

        imc67I 1 Reply Last reply
        2
        • jdaviescoatesJ jdaviescoates

          @imc67 I've not tried yet, but I'm interested in why @girish recommends the kernel reversion as opposed to updating Ubuntu 22.04?

          imc67I Offline
          imc67I Offline
          imc67
          translator
          wrote on last edited by imc67
          #4

          @jdaviescoates said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

          @imc67 I've not tried yet, but I'm interested in why @girish recommends the kernel reversion as opposed to updating Ubuntu 22.04?

          was thinking about that too, but then first you need to update Cloudron to 7.3.2 and that's not stable yet? Or won't even update because of the issues?

          jdaviescoatesJ 1 Reply Last reply
          0
          • imc67I imc67

            @jdaviescoates said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

            @imc67 I've not tried yet, but I'm interested in why @girish recommends the kernel reversion as opposed to updating Ubuntu 22.04?

            was thinking about that too, but then first you need to update Cloudron to 7.3.2 and that's not stable yet? Or won't even update because of the issues?

            jdaviescoatesJ Offline
            jdaviescoatesJ Offline
            jdaviescoates
            wrote on last edited by
            #5

            @imc67 said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

            update Cloudron to 7.3.2 and that's not stable yet?

            I'm already on it.

            @imc67 said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

            Or won't even update because of the issues?

            That could be why I guess, might be risky to update with the issue.

            But still wondering if it might be the best option for me given I'm already on 7.3.2

            I use Cloudron with Gandi & Hetzner

            girishG 1 Reply Last reply
            1
            • girishG girish

              @p44 right. unfortunately, it's all part of the "cron jobs don't work anymore" issue.

              P Offline
              P Offline
              p44
              translator
              wrote on last edited by p44
              #6

              @girish Ok thank's, I applied this https://forum.cloudron.io/topic/8101/fix-for-kernel-bug-in-ubuntu-20-04-causing-various-issues on most of active instances.

              • Should we apply fix to all instances, even those that seemingly don't show any problems?
              • What will be next? When they fix kernel issue, we have to operate again to remove fix that we did?

              Thank's a lot

              girishG 1 Reply Last reply
              1
              • P p44 referenced this topic on
              • jdaviescoatesJ jdaviescoates

                @imc67 said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

                update Cloudron to 7.3.2 and that's not stable yet?

                I'm already on it.

                @imc67 said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

                Or won't even update because of the issues?

                That could be why I guess, might be risky to update with the issue.

                But still wondering if it might be the best option for me given I'm already on 7.3.2

                girishG Offline
                girishG Offline
                girish
                Staff
                wrote on last edited by
                #7

                @jdaviescoates Updating to ubuntu 22.04 is a much riskier endeavor than just downgrading the kernel. Downgrading kernel only takes 5 mins. In fact, just this weekend I upgraded all our servers from Ubuntu 18 to 20 and all of them just completely hosed 😞 Each one failed in different places - one in upgrading cloud-init, another is still stuck in some "conflicting package" and for another I had to switch from DO mirror to canonical's mirror. I have generally not had good experiences with distro upgrades (on the server atleast). On desktop ubuntu, I feel things are better, maybe because I have the PC in front of me and have more control of the boot loader.

                Ubuntu 22 should work fine though with Cloudron 7.3. But note that it requires you to also rebuild all containers because of cgroup v1 to cgroup v2 migration. All this is in the docs but atleast we had 2-3 bug reports of the migration script not working perfectly.

                avatar1024A 1 Reply Last reply
                2
                • P p44

                  @girish Ok thank's, I applied this https://forum.cloudron.io/topic/8101/fix-for-kernel-bug-in-ubuntu-20-04-causing-various-issues on most of active instances.

                  • Should we apply fix to all instances, even those that seemingly don't show any problems?
                  • What will be next? When they fix kernel issue, we have to operate again to remove fix that we did?

                  Thank's a lot

                  girishG Offline
                  girishG Offline
                  girish
                  Staff
                  wrote on last edited by
                  #8

                  @p44 said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

                  Should we apply fix to all instances, even those that seemingly don't show any problems?

                  I think it's best to apply to all instances running that kernel.

                  What will be next? When they fix kernel issue, we have to operate again to remove fix that we did?

                  Yes, I think we can unhold the kernel packages and then it will keep auto updating to latest kernel.

                  jdaviescoatesJ 1 Reply Last reply
                  2
                  • girishG girish

                    @p44 said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

                    Should we apply fix to all instances, even those that seemingly don't show any problems?

                    I think it's best to apply to all instances running that kernel.

                    What will be next? When they fix kernel issue, we have to operate again to remove fix that we did?

                    Yes, I think we can unhold the kernel packages and then it will keep auto updating to latest kernel.

                    jdaviescoatesJ Offline
                    jdaviescoatesJ Offline
                    jdaviescoates
                    wrote on last edited by jdaviescoates
                    #9

                    @girish thanks, makes sense.

                    I guess rather than updating to 22.04 (if people wanted to try that route) it could be safer/ easier to migrate to a fresh install of 22.04?

                    @imc67 fyi I just did the fix above on my primary netcup server (the other one is only running an unloved instance of Uptime Kuma atm), very easy and seems to have gone smoothly.

                    I use Cloudron with Gandi & Hetzner

                    jdaviescoatesJ girishG 2 Replies Last reply
                    4
                    • jdaviescoatesJ jdaviescoates

                      @girish thanks, makes sense.

                      I guess rather than updating to 22.04 (if people wanted to try that route) it could be safer/ easier to migrate to a fresh install of 22.04?

                      @imc67 fyi I just did the fix above on my primary netcup server (the other one is only running an unloved instance of Uptime Kuma atm), very easy and seems to have gone smoothly.

                      jdaviescoatesJ Offline
                      jdaviescoatesJ Offline
                      jdaviescoates
                      wrote on last edited by
                      #10

                      @jdaviescoates said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

                      seems to have gone smoothly.

                      Not so fast, some how my backup mount lost its permission and now I'm unable to remount it

                      I use Cloudron with Gandi & Hetzner

                      1 Reply Last reply
                      1
                      • jdaviescoatesJ jdaviescoates

                        @girish thanks, makes sense.

                        I guess rather than updating to 22.04 (if people wanted to try that route) it could be safer/ easier to migrate to a fresh install of 22.04?

                        @imc67 fyi I just did the fix above on my primary netcup server (the other one is only running an unloved instance of Uptime Kuma atm), very easy and seems to have gone smoothly.

                        girishG Offline
                        girishG Offline
                        girish
                        Staff
                        wrote on last edited by
                        #11

                        @jdaviescoates said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

                        I guess rather than updating to 22.04 (if people wanted to try that route) it could be safer/ easier to migrate to a fresh install of 22.04?

                        yes, definitely.

                        martinkbsM 1 Reply Last reply
                        1
                        • girishG girish forked this topic on
                        • jdaviescoatesJ jdaviescoates referenced this topic on
                        • girishG girish

                          This fix is not needed anymore. Ubuntu has released 5.4.0-135-generic

                          Cloudron enables automatic ubuntu security updates. Roughly around 2022-11-17, the linux kernel was updated to 5.4.0-132-generic . You can find the automatic updates log in /var/log/apt/history.log. This kernel has a bug causing various things like containerd, prometheus node exporter etc to fail. On Cloudron, this manifests itself as:

                          • automatic updates appear to get stuck in 'cleaning up old install"
                          • cron jobs don't work anymore
                          • file permissions inside containers become incorrect

                          For the moment, it's best to revert to the previous kernel 5.4.0-131-generic. How you do this, depends on your VPS provider. Some VPS providers allow you to change the kernel via their control panels.

                          Please be careful with instructions below. You might have to fine tune it based on your setup/provider.

                          Many of the modern provider will just use Grub 2 as the kernel (digitalocean, linode, to name a few). On such VPS, please change the kernel as follows:

                          • Highly recommend taking a snapshot of the server, in case something goes wrong.
                          • SSH into the server
                          • apt install linux-image-5.4.0-131-generic linux-modules-extra-5.4.0-131-generic
                          • apt-mark hold linux-generic linux-image-generic linux-headers-generic
                          • Edit /etc/default/grub . Find the line GRUB_DEFAULT=0. Change this to GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 5.4.0-131-generic" . Important you get this line right, otherwise your server may not boot!
                          • update-grub
                          • reboot
                          • After reboot, uname -nar will say 5.4.0-131-generic.

                          To reverse the above changes:

                          • apt-mark unhold linux-generic linux-image-generic linux-headers-generic
                          • unattended-upgrade -d - when running this you will see new kernel 5.4.0-135-generic is getting installed.
                          • Edit /etc/default/grub . Change the line to GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 5.4.0-135-generic".
                          • update-grub
                          • reboot
                          • After reboot, uname -nar will say 5.4.0-135-generic
                          • apt remove linux-image-5.4.0-131-generic linux-modules-extra-5.4.0-131-generic - to remove the old kernel
                          • Edit /etc/default/grub . Change the line to `GRUB_DEFAULT=0
                          • update-grub

                          Related threads:

                          • https://forum.cloudron.io/topic/8096/docker-needs-to-be-restarted-often-due-to-fail-app-updates
                          • https://forum.cloudron.io/topic/8032/apps-stuck-updating-cleaning-up-old-install-even-after-stopping-restarting-task
                          P Offline
                          P Offline
                          p44
                          translator
                          wrote on last edited by p44
                          #12

                          @girish Hello Girish,

                          after fix «automatic updates appear to get stuck in 'cleaning up old install"» has been solved, but it seems "cron jobs don't work anymore" problem is still there...

                          Do you have any other feedback on this issue?

                          jdaviescoatesJ 1 Reply Last reply
                          0
                          • P p44

                            @girish Hello Girish,

                            after fix «automatic updates appear to get stuck in 'cleaning up old install"» has been solved, but it seems "cron jobs don't work anymore" problem is still there...

                            Do you have any other feedback on this issue?

                            jdaviescoatesJ Offline
                            jdaviescoatesJ Offline
                            jdaviescoates
                            wrote on last edited by
                            #13

                            @p44 said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

                            "cron jobs don't work anymore" problem is still there.

                            Seems to have gone for me. Previously my Nextclouds were giving me warnings about that, but they aren't doing that anymore.

                            I use Cloudron with Gandi & Hetzner

                            P 1 Reply Last reply
                            2
                            • jdaviescoatesJ jdaviescoates

                              @p44 said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

                              "cron jobs don't work anymore" problem is still there.

                              Seems to have gone for me. Previously my Nextclouds were giving me warnings about that, but they aren't doing that anymore.

                              P Offline
                              P Offline
                              p44
                              translator
                              wrote on last edited by
                              #14

                              @jdaviescoates Thank's a lot, I'll do more accurate tests ... it seems only few cron jobs are executed, in external cron panel

                              1 Reply Last reply
                              0
                              • P Offline
                                P Offline
                                privsec
                                wrote on last edited by
                                #15

                                Is this a need to be done?

                                Can this get automated by Cloudron?

                                girishG 1 Reply Last reply
                                0
                                • P privsec

                                  Is this a need to be done?

                                  Can this get automated by Cloudron?

                                  girishG Offline
                                  girishG Offline
                                  girish
                                  Staff
                                  wrote on last edited by
                                  #16

                                  @privsec It's not automated. I think automating kernel updates would be quite a reach for us since we don't control or have access to the hardware. We have to rely on ubuntu/debian to get their testing right here.

                                  1 Reply Last reply
                                  3
                                  • girishG girish

                                    @jdaviescoates Updating to ubuntu 22.04 is a much riskier endeavor than just downgrading the kernel. Downgrading kernel only takes 5 mins. In fact, just this weekend I upgraded all our servers from Ubuntu 18 to 20 and all of them just completely hosed 😞 Each one failed in different places - one in upgrading cloud-init, another is still stuck in some "conflicting package" and for another I had to switch from DO mirror to canonical's mirror. I have generally not had good experiences with distro upgrades (on the server atleast). On desktop ubuntu, I feel things are better, maybe because I have the PC in front of me and have more control of the boot loader.

                                    Ubuntu 22 should work fine though with Cloudron 7.3. But note that it requires you to also rebuild all containers because of cgroup v1 to cgroup v2 migration. All this is in the docs but atleast we had 2-3 bug reports of the migration script not working perfectly.

                                    avatar1024A Offline
                                    avatar1024A Offline
                                    avatar1024
                                    wrote on last edited by
                                    #17

                                    @girish said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

                                    I upgraded all our servers from Ubuntu 18 to 20 and all of them just completely hosed

                                    Interesting.... I upgraded one server from 16 > 18 > 20 and two more servers from 18 > 20 using the Cloudron guides and never had a problem. I was going to upgrade to 22 thinking it'll all be easy but a little more unsure now.

                                    girishG 1 Reply Last reply
                                    1
                                    • avatar1024A avatar1024

                                      @girish said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

                                      I upgraded all our servers from Ubuntu 18 to 20 and all of them just completely hosed

                                      Interesting.... I upgraded one server from 16 > 18 > 20 and two more servers from 18 > 20 using the Cloudron guides and never had a problem. I was going to upgrade to 22 thinking it'll all be easy but a little more unsure now.

                                      girishG Offline
                                      girishG Offline
                                      girish
                                      Staff
                                      wrote on last edited by
                                      #18

                                      @avatar1024 I think it's some issue with the DO apt mirrors. Something is out of rsync. I test out upgrading in vultr/linode and they seem perfect.

                                      1 Reply Last reply
                                      1
                                      • BrutalBirdieB BrutalBirdie referenced this topic on
                                      • girishG Offline
                                        girishG Offline
                                        girish
                                        Staff
                                        wrote on last edited by
                                        #19

                                        Ubuntu released a new kenel with the fix 5.4.0-135.152 - https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1996678/comments/28 . I don't know if this kernel arrives as a security update.

                                        P 1 Reply Last reply
                                        3
                                        • nebulonN Offline
                                          nebulonN Offline
                                          nebulon
                                          Staff
                                          wrote on last edited by
                                          #20

                                          At least on vultr Ubuntu 20.04 repository mirrors, the new fixed kernel is already available via security updates. You can check if this is the same in your instance by running:

                                          apt-get update && apt list --upgradable | grep "\-security"
                                          

                                          If it lists linux-generic/focal-updates,focal-security 5.4.0.135.133 amd64 [upgradable from: 5.4.0.132.132] then you have to unhold the previously hold packages and eventually it will normally update:

                                          apt-mark unhold linux-generic linux-image-generic linux-headers-generic
                                          
                                          P 1 Reply Last reply
                                          4
                                          Reply
                                          • Reply as topic
                                          Log in to reply
                                          • Oldest to Newest
                                          • Newest to Oldest
                                          • Most Votes


                                          • Login

                                          • Don't have an account? Register

                                          • Login or register to search.
                                          • First post
                                            Last post
                                          0
                                          • Categories
                                          • Recent
                                          • Tags
                                          • Popular
                                          • Bookmarks
                                          • Search