Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    Solved Cloudron update failing

    Support
    update
    3
    13
    376
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • G
      gerben last edited by girish

      Updating Cloudron seems to have been failing for me, for a while already. I just tried run an update manually, with backups disabled, from version 6.1.2 to 6.2.7 (not sure why it does not try to jump further than that); it simply stops after some seconds, with this in the log:

      Jan 13 14:48:41 box:locker Acquired : box_update
      Jan 13 14:48:41 box:tasks startTask - starting task 3371. logs at /home/yellowtent/platformdata/logs/tasks/3371.log
      Jan 13 14:48:41 box:shell startTask spawn: /usr/bin/sudo -S -E /home/yellowtent/box/src/scripts/starttask.sh 3371 /home/yellowtent/platformdata/logs/tasks/3371.log 15 400
      Jan 13 14:48:41 box:shell startTask (stdout): Running as unit: box-task-3371.service
      Jan 13 14:48:49 box:shell startTask (stdout): Finished with result: exit-code
      processes terminated with: code=exited/status=50
      runtime: 8.688s
      Jan 13 14:48:49 box:shell startTask code: 50, signal: null
      Jan 13 14:48:49 box:tasks startTask: 3371 completed with code 50 and signal null
      Jan 13 14:48:49 box:locker Released : box_update
      Jan 13 14:48:49 box:updater Update failed with error { stack:
      'BoxError: update exited with code 1 signal null\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:66:17)\n at ChildProcess.emit (events.js:198:13)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)',
      name: 'BoxError',
      reason: 'Spawn Error',
      details: {},
      message: 'update exited with code 1 signal null',
      code: 1,
      signal: null }
      Jan 13 14:48:49 box:tasks startTask: 3371 done
      

      Any suggestions what to do, where to look? I could just reinstall from scratch if that seems the easier solution. I seem to be having other issues too, with backups not being cleaned up, and apps not automatically updating either; perhaps there is a deeper issue underneath all those quirks.

      1 Reply Last reply Reply Quote 0
      • nebulon
        nebulon Staff last edited by

        are there more error details at /home/yellowtent/platformdata/logs/tasks/3371.log ?

        Also skipping backup might not be a good idea, if this is a production system.

        1 Reply Last reply Reply Quote 0
        • G
          gerben last edited by

          Ah yes I did not notice the task has its own log file, I overlooked that line.

          From that log (after some messages about downloading):

          2022-01-13T13:48:49.786Z box:tasks 3371: {"percent":70,"message":"Installing update"}
          2022-01-13T13:48:49.786Z box:shell update spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/update.sh /tmp/box-3242447597
          2022-01-13T13:48:49.803Z box:shell update (stdout): Updating Cloudron with /tmp/box-3242447597
          => reset service cloudron-updater status (of previous update)
          
          2022-01-13T13:48:49.809Z box:shell update (stdout): => Run installer.sh as cloudron-updater.
          
          2022-01-13T13:48:49.818Z box:shell update (stdout): => starting service (ubuntu 18.04) cloudron-updater. see logs at /home/yellowtent/platformdata/logs/updater/cloudron-updater-2022-01-13_13-48-49.log
          
          2022-01-13T13:48:49.826Z box:shell update (stdout): Failed to start transient service unit: Unit cloudron-updater.service already exists.
          
          2022-01-13T13:48:49.827Z box:shell update (stdout): Failed to install cloudron. See log for details
          
          2022-01-13T13:48:49.828Z box:shell update code: 1, signal: null
          2022-01-13T13:48:49.829Z box:taskworker Task took 7.718 seconds
          2022-01-13T13:48:49.829Z box:tasks setCompleted - 3371: {"result":null,"error":{"stack":"BoxError: update exited with code 1 signal null\n    at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:66:17)\n    at ChildProcess.emit (events.js:198:13)\n    at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)","name":"BoxError","reason":"Spawn Error","details":{},"message":"update exited with code 1 signal null","code":1,"signal":null}}
          2022-01-13T13:48:49.829Z box:tasks 3371: {"percent":100,"result":null,"error":{"stack":"BoxError: update exited with code 1 signal null\n    at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:66:17)\n    at ChildProcess.emit (events.js:198:13)\n    at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)","name":"BoxError","reason":"Spawn Error","details":{},"message":"update exited with code 1 signal null","code":1,"signal":null}}
          

          The main error being “Failed to start transient service unit: Unit cloudron-updater.service already exists.”

          Before you ask, the mentioned /home/yellowtent/platformdata/logs/updater/cloudron-updater-2022-01-13_13-48-49.log file does not exist.

          Not sure it matters that it mentions “(ubuntu 18.04) cloudron updater” while the server runs Ubuntu 20.04.1 LTS.

          I guess the issue is with this service?:

          # service cloudron-updater status
          ● cloudron-updater.service - /tmp/box-2930230092/scripts/installer.sh
               Loaded: loaded (/run/systemd/transient/cloudron-updater.service; transient)
            Transient: yes
               Active: active (running) since Tue 2021-11-02 05:00:56 UTC; 2 months 11 days ago
             Main PID: 30747 (installer.sh)
                Tasks: 2 (limit: 4660)
               Memory: 16.8M
               CGroup: /system.slice/cloudron-updater.service
                       ├─  30747 /bin/bash /tmp/box-2930230092/scripts/installer.sh
                       └─3797894 sleep 1
          
          G 1 Reply Last reply Reply Quote 0
          • G
            gerben @gerben last edited by

            So it looks like the updater service has been running some months. The latest updater log seems to match the date it started (updater/cloudron-updater-2021-11-02_05-00-56.log), and it is a 6 GB file that looks like it repeats this bunch of lines:

            dpkg: dependency problems prevent configuration of linux-image-generic:
             linux-image-generic depends on linux-modules-extra-5.4.0-59-generic; however:
              Package linux-modules-extra-5.4.0-59-generic is not installed.
            
            dpkg: error processing package linux-image-generic (--configure):
             dependency problems - leaving unconfigured
            dpkg: dependency problems prevent configuration of linux-generic:
             linux-generic depends on linux-image-generic (= 5.4.0.59.62); however:
              Package linux-image-generic is not configured yet.
            
            dpkg: error processing package linux-generic (--configure):
             dependency problems - leaving unconfigured
            dpkg: dependency problems prevent configuration of linux-image-5.4.0-59-generic:
             linux-image-5.4.0-59-generic depends on linux-modules-5.4.0-59-generic; however:
              Package linux-modules-5.4.0-59-generic is not installed.
            
            dpkg: error processing package linux-image-5.4.0-59-generic (--configure):
             dependency problems - leaving unconfigured
            Errors were encountered while processing:
             linux-image-generic
             linux-generic
             linux-image-5.4.0-59-generic
            2022-01-13T00:00:17 ==> installer: Failed to fix packages. Retry
            

            I’ll try stop the service and try again, and see what happens.

            G 1 Reply Last reply Reply Quote 0
            • G
              gerben @gerben last edited by

              Ok.. the underlying cause seems to have been that my /lib/modules is a 100MB tmpfs filesystem, on which dpkg tried to unpack 250MB of files. I don’t know how/why this happened, but I suppose it is not Cloudron’s fault either. (I managed to solve things by running umount /lib/modules so I could run apt --fix-broken install without the disk space error; I’m somewhat surprised the machine managed to reboot after that)

              Anyhow, the part that is relevant for Cloudron seems to be that an issue in ubuntu/dpkg/whatever while updating things can apparently end up in an infinite loop that blocks future updates while creating gigabytes of log files; without feedback to the user that something is going wrong, let alone what is going wrong. Perhaps it could already help to have a time-limit on the updater service, and/or making dpkg stop rather than keep retrying, and then the UI can report that updating failed?

              G girish 2 Replies Last reply Reply Quote 0
              • G
                gerben @gerben last edited by gerben

                And another request from this case: could it be possible to have the updater jump forward multiple versions at once? My “update available” just shows the next version, then after updating I click ‘check for updates’ and it tells me there is yet another one, and so forth.. will take me an hour of clicking to get up to date again. I noticed the same thing seems to be the case when updating an individual app, rather cumbersome!

                In any case, thanks still for the swift response and keep up the good work! 🙂

                girish 1 Reply Last reply Reply Quote 0
                • girish
                  girish Staff @gerben last edited by

                  @gerben mm, in theory, it should have failed after some tries. maybe there is an infinite try loop somewhere. checking

                  1 Reply Last reply Reply Quote 0
                  • girish
                    girish Staff last edited by

                    Indeed, the docker install logic was not cleaned up properly when we cleaned up other package installs. fixed now - https://git.cloudron.io/cloudron/box/-/commit/09d3d258b61ba338c0fadc88c0418d8bcc77f528 . Thanks for investigating!

                    G 1 Reply Last reply Reply Quote 0
                    • girish
                      girish Staff @gerben last edited by

                      @gerben said in Cloudron update failing:

                      And another request from this case: could it be possible to have the updater jump forward multiple versions at once

                      We use a Continuous delivery/Rolling update model. We designed Cloudron to be more closer to SaaS where things are always up to date (unlike traditional enterprise selfhosted software which have LTS releases). In general, costs of supporting LTS style is too high if you make frequent releases like we do with Cloudron. But yes, as you found out, this model also does not work well for those who don't update regularly/auto-update.

                      G 1 Reply Last reply Reply Quote 0
                      • G
                        gerben @girish last edited by

                        @girish thanks for the quick reaction!

                        I understand you aim for the rolling-update approach, and I can also imagine that skipping some versions could lead to issues. But even if this stays this way, a tweak (mostly UI) that would let me say “run all missing updates one after another”, would save a lot of waiting and clicking for those cases where a version of cloudron or some app was held back. I see this has been argued for before, with more examples for why this may not be “a corner case”. I hope you will consider the option!

                        girish 1 Reply Last reply Reply Quote 0
                        • girish
                          girish Staff @gerben last edited by

                          @gerben ah yes, an option/button to "update to latest" which continuously applies updates, instead of having to manually keep clicking, makes much sense. Do you think you can open a request at https://forum.cloudron.io/category/97/feature-requests ?

                          G 1 Reply Last reply Reply Quote 0
                          • Referenced by  G gerben 
                          • G
                            gerben @girish last edited by

                            @girish done! https://forum.cloudron.io/topic/6333/update-to-latest-button-option

                            1 Reply Last reply Reply Quote 1
                            • G
                              gerben @girish last edited by

                              @girish said in Cloudron update failing:

                              Indeed, the docker install logic was not cleaned up properly when we cleaned up other package installs. fixed now - https://git.cloudron.io/cloudron/box/-/commit/09d3d258b61ba338c0fadc88c0418d8bcc77f528 . Thanks for investigating!

                              Perhaps that fixes a similar bug, but just for clarity, note that I was using an old version Cloudron (6.1.2), and got stuck in this other loop that looks like it has already been removed from the script between version 6.2 and 6.3.

                              while ! dpkg --force-confold --configure -a; do
                                  log "Failed to fix packages. Retry"
                                  sleep 1
                              done
                              

                              That “Failed to fix packages” is the message that appeared again and again in my infinite logs:

                              2022-01-13T17:05:53 ==> installer: Updating from 6.1.2 to 6.2.7
                              2022-01-13T17:05:53 ==> installer: updating docker
                              2022-01-13T17:05:58 ==> installer: Waiting for all dpkg tasks to finish...
                              dpkg: dependency problems prevent configuration of linux-image-generic:
                               linux-image-generic depends on linux-modules-extra-5.4.0-59-generic; however:
                                Package linux-modules-extra-5.4.0-59-generic is not installed.
                              
                              dpkg: error processing package linux-image-generic (--configure):
                               dependency problems - leaving unconfigured
                              dpkg: dependency problems prevent configuration of linux-generic:
                               linux-generic depends on linux-image-generic (= 5.4.0.59.62); however:
                                Package linux-image-generic is not configured yet.
                              
                              dpkg: error processing package linux-generic (--configure):
                               dependency problems - leaving unconfigured
                              dpkg: dependency problems prevent configuration of linux-image-5.4.0-59-generic:
                               linux-image-5.4.0-59-generic depends on linux-modules-5.4.0-59-generic; however:
                                Package linux-modules-5.4.0-59-generic is not installed.
                              
                              dpkg: error processing package linux-image-5.4.0-59-generic (--configure):
                               dependency problems - leaving unconfigured
                              Errors were encountered while processing:
                               linux-image-generic
                               linux-generic
                               linux-image-5.4.0-59-generic
                              2022-01-13T17:05:58 ==> installer: Failed to fix packages. Retry
                              dpkg: dependency problems prevent configuration of linux-image-generic:
                              …
                              …
                              
                              1 Reply Last reply Reply Quote 1
                              • First post
                                Last post
                              Powered by NodeBB