Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Support
  3. Fix for kernel bug in Ubuntu 20.04 causing various issues

Fix for kernel bug in Ubuntu 20.04 causing various issues

Scheduled Pinned Locked Moved Solved Support
44 Posts 8 Posters 6.3k Views 9 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • jdaviescoatesJ jdaviescoates

    @girish thanks, makes sense.

    I guess rather than updating to 22.04 (if people wanted to try that route) it could be safer/ easier to migrate to a fresh install of 22.04?

    @imc67 fyi I just did the fix above on my primary netcup server (the other one is only running an unloved instance of Uptime Kuma atm), very easy and seems to have gone smoothly.

    girishG Offline
    girishG Offline
    girish
    Staff
    wrote on last edited by
    #11

    @jdaviescoates said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

    I guess rather than updating to 22.04 (if people wanted to try that route) it could be safer/ easier to migrate to a fresh install of 22.04?

    yes, definitely.

    martinkbsM 1 Reply Last reply
    1
    • girishG girish forked this topic on
    • jdaviescoatesJ jdaviescoates referenced this topic on
    • girishG girish

      This fix is not needed anymore. Ubuntu has released 5.4.0-135-generic

      Cloudron enables automatic ubuntu security updates. Roughly around 2022-11-17, the linux kernel was updated to 5.4.0-132-generic . You can find the automatic updates log in /var/log/apt/history.log. This kernel has a bug causing various things like containerd, prometheus node exporter etc to fail. On Cloudron, this manifests itself as:

      • automatic updates appear to get stuck in 'cleaning up old install"
      • cron jobs don't work anymore
      • file permissions inside containers become incorrect

      For the moment, it's best to revert to the previous kernel 5.4.0-131-generic. How you do this, depends on your VPS provider. Some VPS providers allow you to change the kernel via their control panels.

      Please be careful with instructions below. You might have to fine tune it based on your setup/provider.

      Many of the modern provider will just use Grub 2 as the kernel (digitalocean, linode, to name a few). On such VPS, please change the kernel as follows:

      • Highly recommend taking a snapshot of the server, in case something goes wrong.
      • SSH into the server
      • apt install linux-image-5.4.0-131-generic linux-modules-extra-5.4.0-131-generic
      • apt-mark hold linux-generic linux-image-generic linux-headers-generic
      • Edit /etc/default/grub . Find the line GRUB_DEFAULT=0. Change this to GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 5.4.0-131-generic" . Important you get this line right, otherwise your server may not boot!
      • update-grub
      • reboot
      • After reboot, uname -nar will say 5.4.0-131-generic.

      To reverse the above changes:

      • apt-mark unhold linux-generic linux-image-generic linux-headers-generic
      • unattended-upgrade -d - when running this you will see new kernel 5.4.0-135-generic is getting installed.
      • Edit /etc/default/grub . Change the line to GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 5.4.0-135-generic".
      • update-grub
      • reboot
      • After reboot, uname -nar will say 5.4.0-135-generic
      • apt remove linux-image-5.4.0-131-generic linux-modules-extra-5.4.0-131-generic - to remove the old kernel
      • Edit /etc/default/grub . Change the line to `GRUB_DEFAULT=0
      • update-grub

      Related threads:

      • https://forum.cloudron.io/topic/8096/docker-needs-to-be-restarted-often-due-to-fail-app-updates
      • https://forum.cloudron.io/topic/8032/apps-stuck-updating-cleaning-up-old-install-even-after-stopping-restarting-task
      P Offline
      P Offline
      p44
      translator
      wrote on last edited by p44
      #12

      @girish Hello Girish,

      after fix Β«automatic updates appear to get stuck in 'cleaning up old install"Β» has been solved, but it seems "cron jobs don't work anymore" problem is still there...

      Do you have any other feedback on this issue?

      jdaviescoatesJ 1 Reply Last reply
      0
      • P p44

        @girish Hello Girish,

        after fix Β«automatic updates appear to get stuck in 'cleaning up old install"Β» has been solved, but it seems "cron jobs don't work anymore" problem is still there...

        Do you have any other feedback on this issue?

        jdaviescoatesJ Offline
        jdaviescoatesJ Offline
        jdaviescoates
        wrote on last edited by
        #13

        @p44 said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

        "cron jobs don't work anymore" problem is still there.

        Seems to have gone for me. Previously my Nextclouds were giving me warnings about that, but they aren't doing that anymore.

        I use Cloudron with Gandi & Hetzner

        P 1 Reply Last reply
        2
        • jdaviescoatesJ jdaviescoates

          @p44 said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

          "cron jobs don't work anymore" problem is still there.

          Seems to have gone for me. Previously my Nextclouds were giving me warnings about that, but they aren't doing that anymore.

          P Offline
          P Offline
          p44
          translator
          wrote on last edited by
          #14

          @jdaviescoates Thank's a lot, I'll do more accurate tests ... it seems only few cron jobs are executed, in external cron panel

          1 Reply Last reply
          0
          • P Offline
            P Offline
            privsec
            wrote on last edited by
            #15

            Is this a need to be done?

            Can this get automated by Cloudron?

            girishG 1 Reply Last reply
            0
            • P privsec

              Is this a need to be done?

              Can this get automated by Cloudron?

              girishG Offline
              girishG Offline
              girish
              Staff
              wrote on last edited by
              #16

              @privsec It's not automated. I think automating kernel updates would be quite a reach for us since we don't control or have access to the hardware. We have to rely on ubuntu/debian to get their testing right here.

              1 Reply Last reply
              3
              • girishG girish

                @jdaviescoates Updating to ubuntu 22.04 is a much riskier endeavor than just downgrading the kernel. Downgrading kernel only takes 5 mins. In fact, just this weekend I upgraded all our servers from Ubuntu 18 to 20 and all of them just completely hosed 😞 Each one failed in different places - one in upgrading cloud-init, another is still stuck in some "conflicting package" and for another I had to switch from DO mirror to canonical's mirror. I have generally not had good experiences with distro upgrades (on the server atleast). On desktop ubuntu, I feel things are better, maybe because I have the PC in front of me and have more control of the boot loader.

                Ubuntu 22 should work fine though with Cloudron 7.3. But note that it requires you to also rebuild all containers because of cgroup v1 to cgroup v2 migration. All this is in the docs but atleast we had 2-3 bug reports of the migration script not working perfectly.

                avatar1024A Offline
                avatar1024A Offline
                avatar1024
                wrote on last edited by
                #17

                @girish said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

                I upgraded all our servers from Ubuntu 18 to 20 and all of them just completely hosed

                Interesting.... I upgraded one server from 16 > 18 > 20 and two more servers from 18 > 20 using the Cloudron guides and never had a problem. I was going to upgrade to 22 thinking it'll all be easy but a little more unsure now.

                girishG 1 Reply Last reply
                1
                • avatar1024A avatar1024

                  @girish said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

                  I upgraded all our servers from Ubuntu 18 to 20 and all of them just completely hosed

                  Interesting.... I upgraded one server from 16 > 18 > 20 and two more servers from 18 > 20 using the Cloudron guides and never had a problem. I was going to upgrade to 22 thinking it'll all be easy but a little more unsure now.

                  girishG Offline
                  girishG Offline
                  girish
                  Staff
                  wrote on last edited by
                  #18

                  @avatar1024 I think it's some issue with the DO apt mirrors. Something is out of rsync. I test out upgrading in vultr/linode and they seem perfect.

                  1 Reply Last reply
                  1
                  • BrutalBirdieB BrutalBirdie referenced this topic on
                  • girishG Offline
                    girishG Offline
                    girish
                    Staff
                    wrote on last edited by
                    #19

                    Ubuntu released a new kenel with the fix 5.4.0-135.152 - https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1996678/comments/28 . I don't know if this kernel arrives as a security update.

                    P 1 Reply Last reply
                    3
                    • nebulonN Away
                      nebulonN Away
                      nebulon
                      Staff
                      wrote on last edited by
                      #20

                      At least on vultr Ubuntu 20.04 repository mirrors, the new fixed kernel is already available via security updates. You can check if this is the same in your instance by running:

                      apt-get update && apt list --upgradable | grep "\-security"
                      

                      If it lists linux-generic/focal-updates,focal-security 5.4.0.135.133 amd64 [upgradable from: 5.4.0.132.132] then you have to unhold the previously hold packages and eventually it will normally update:

                      apt-mark unhold linux-generic linux-image-generic linux-headers-generic
                      
                      P 1 Reply Last reply
                      4
                      • girishG girish

                        Ubuntu released a new kenel with the fix 5.4.0-135.152 - https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1996678/comments/28 . I don't know if this kernel arrives as a security update.

                        P Offline
                        P Offline
                        p44
                        translator
                        wrote on last edited by
                        #21

                        @girish Hello Girish, did you check if this kernel arrives as a security update?

                        Can we unhold previously hold packages?

                        Thank's a lot πŸ™

                        girishG 1 Reply Last reply
                        0
                        • P p44

                          @girish Hello Girish, did you check if this kernel arrives as a security update?

                          Can we unhold previously hold packages?

                          Thank's a lot πŸ™

                          girishG Offline
                          girishG Offline
                          girish
                          Staff
                          wrote on last edited by
                          #22

                          @p44 this seems to have come to our servers atleast. So, yes, go ahead an unmark the holds.

                          Start-Date: 2022-12-03  06:10:57
                          Commandline: /usr/bin/unattended-upgrade
                          Install: linux-image-5.4.0-135-generic:amd64 (5.4.0-135.152, automatic), linux-headers-5.4.0-135:amd64 (5.4.0-135.152, automatic), linux-modules-extra-5.4.0-135-generic:amd64 (5.4.0-135.152, automatic), linux-modules-5.4.0-135-generic:amd64 (5.4.0-135.152, automatic), linux-headers-5.4.0-135-generic:amd64 (5.4.0-135.152, automatic)
                          Upgrade: linux-headers-generic:amd64 (5.4.0.132.132, 5.4.0.135.133), linux-image-generic:amd64 (5.4.0.132.132, 5.4.0.135.133), linux-generic:amd64 (5.4.0.132.132, 5.4.0.135.133)
                          End-Date: 2022-12-03  06:12:14
                          
                          P 1 Reply Last reply
                          1
                          • girishG girish unpinned this topic on
                          • girishG girish marked this topic as a question on
                          • girishG girish has marked this topic as solved on
                          • girishG girish

                            @p44 this seems to have come to our servers atleast. So, yes, go ahead an unmark the holds.

                            Start-Date: 2022-12-03  06:10:57
                            Commandline: /usr/bin/unattended-upgrade
                            Install: linux-image-5.4.0-135-generic:amd64 (5.4.0-135.152, automatic), linux-headers-5.4.0-135:amd64 (5.4.0-135.152, automatic), linux-modules-extra-5.4.0-135-generic:amd64 (5.4.0-135.152, automatic), linux-modules-5.4.0-135-generic:amd64 (5.4.0-135.152, automatic), linux-headers-5.4.0-135-generic:amd64 (5.4.0-135.152, automatic)
                            Upgrade: linux-headers-generic:amd64 (5.4.0.132.132, 5.4.0.135.133), linux-image-generic:amd64 (5.4.0.132.132, 5.4.0.135.133), linux-generic:amd64 (5.4.0.132.132, 5.4.0.135.133)
                            End-Date: 2022-12-03  06:12:14
                            
                            P Offline
                            P Offline
                            p44
                            translator
                            wrote on last edited by
                            #23

                            @girish Ok, thank's a lot πŸ™

                            1 Reply Last reply
                            0
                            • nebulonN nebulon

                              At least on vultr Ubuntu 20.04 repository mirrors, the new fixed kernel is already available via security updates. You can check if this is the same in your instance by running:

                              apt-get update && apt list --upgradable | grep "\-security"
                              

                              If it lists linux-generic/focal-updates,focal-security 5.4.0.135.133 amd64 [upgradable from: 5.4.0.132.132] then you have to unhold the previously hold packages and eventually it will normally update:

                              apt-mark unhold linux-generic linux-image-generic linux-headers-generic
                              
                              P Offline
                              P Offline
                              p44
                              translator
                              wrote on last edited by p44
                              #24

                              @nebulon Should we have to reverse also this step?

                              • Edit /etc/default/grub . Find the line GRUB_DEFAULT=0. Change this to GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 5.4.0-131-generic" . Important you get this line right, otherwise your server may not boot
                              girishG 1 Reply Last reply
                              1
                              • P p44

                                @nebulon Should we have to reverse also this step?

                                • Edit /etc/default/grub . Find the line GRUB_DEFAULT=0. Change this to GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 5.4.0-131-generic" . Important you get this line right, otherwise your server may not boot
                                girishG Offline
                                girishG Offline
                                girish
                                Staff
                                wrote on last edited by
                                #25

                                @p44 yes

                                P 1 Reply Last reply
                                1
                                • girishG girish

                                  @p44 yes

                                  P Offline
                                  P Offline
                                  p44
                                  translator
                                  wrote on last edited by
                                  #26

                                  @girish @nebulon please can you post here all the whole process to reverse this fix?

                                  P 1 Reply Last reply
                                  2
                                  • P p44

                                    @girish @nebulon please can you post here all the whole process to reverse this fix?

                                    P Offline
                                    P Offline
                                    privsec
                                    wrote on last edited by
                                    #27

                                    @p44 said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

                                    @girish @nebulon please can you post here all the whole process to reverse this fix?

                                    Yes please, the whole process.

                                    P 1 Reply Last reply
                                    2
                                    • P privsec

                                      @p44 said in Fix for kernel bug in Ubuntu 20.04 causing various issues:

                                      @girish @nebulon please can you post here all the whole process to reverse this fix?

                                      Yes please, the whole process.

                                      P Offline
                                      P Offline
                                      p44
                                      translator
                                      wrote on last edited by
                                      #28

                                      Hello @girish and @nebulon, when you have time can you take a look to this? Thank's πŸ™

                                      girishG 1 Reply Last reply
                                      0
                                      • P p44

                                        Hello @girish and @nebulon, when you have time can you take a look to this? Thank's πŸ™

                                        girishG Offline
                                        girishG Offline
                                        girish
                                        Staff
                                        wrote on last edited by
                                        #29

                                        @p44 I put in instructions to reverse in the original post.

                                        P O 2 Replies Last reply
                                        1
                                        • girishG girish

                                          @p44 I put in instructions to reverse in the original post.

                                          P Offline
                                          P Offline
                                          p44
                                          translator
                                          wrote on last edited by
                                          #30

                                          @girish Thank's a lot πŸ‘πŸ‘πŸ‘

                                          1 Reply Last reply
                                          0
                                          Reply
                                          • Reply as topic
                                          Log in to reply
                                          • Oldest to Newest
                                          • Newest to Oldest
                                          • Most Votes


                                          • Login

                                          • Don't have an account? Register

                                          • Login or register to search.
                                          • First post
                                            Last post
                                          0
                                          • Categories
                                          • Recent
                                          • Tags
                                          • Popular
                                          • Bookmarks
                                          • Search