Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    Solved 6.3.3 a few quirks

    Support
    update
    8
    27
    962
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • luckow
      luckow translator last edited by luckow

      From the log:

      POST /api/v1/tasks/3302/stop 409 Conflict task is not active 12.361 ms - 61
      Jun 25 19:54:06 box:scheduler sync: error creating jobs (HTTP code 400) unexpected - No such container: def7655c9e0a287873abde909aea7f722c6795a26d91b4ab355d2f3106043ea8 
      

      Symptom: Configuring (Queued) - Queued in a WordPress app after updating one Cloudron with 39 38 apps. Press the x for cancellation does not have any effect. No backup tasks or anything else. Fun fact: a restart does not work (in the sense of pressing the button in the dashboard)

      Tomorrow I will try to reboot via shell.

      Pronouns: he/him | Primary language: German

      girish 1 Reply Last reply Reply Quote 0
      • girish
        girish Staff @luckow last edited by

        @luckow the second line can be ignored, it's not related to the first. the stop button does not work immediately after an update because the update is sort of still "happening" in the background (because we want user to see the dashboard as early as possible). after some time, do you see all the configuring things moving ahead? only at that point will the stop button work. I guess "queued" is a geeky term but it means that at some point a task will be started.

        luckow 1 Reply Last reply Reply Quote 0
        • luckow
          luckow translator @girish last edited by

          @girish
          7 hours ago THEADMIN - 1.1.1.1 Cloudron update to version 6.3.2 was started

          Pronouns: he/him | Primary language: German

          luckow 1 Reply Last reply Reply Quote 0
          • luckow
            luckow translator @luckow last edited by

            Waiting for (let's say) 1 hour for updating/configuring all 39 app is ok. 7 hours later, it looks like "something is broken". This is the reason why I want to wait until tomorrow and restart the instance via shell.

            Pronouns: he/him | Primary language: German

            girish 1 Reply Last reply Reply Quote 0
            • girish
              girish Staff @luckow last edited by

              @luckow Most likely because the backup took a very long time. Could it be?

              luckow 1 Reply Last reply Reply Quote 0
              • luckow
                luckow translator @girish last edited by

                @girish No. As I said in the first comment: no backup task. This was my first idea from a Cloudron administrator's point of view 🙂
                e209c6d0-87d7-4a40-9c9a-7d3cd713a18f-image.png

                Pronouns: he/him | Primary language: German

                girish 1 Reply Last reply Reply Quote 0
                • girish
                  girish Staff @luckow last edited by

                  @luckow That is strange.. How did the update succeed with no backup 🤔

                  luckow 1 Reply Last reply Reply Quote 0
                  • luckow
                    luckow translator @girish last edited by

                    @girish easy. skip backup 🙂 There is a checkbox.

                    Pronouns: he/him | Primary language: German

                    girish 1 Reply Last reply Reply Quote 1
                    • girish
                      girish Staff @luckow last edited by

                      @luckow Thanks for access to the server, I found the issue but not sure why it happens yet. The box code temporarily lost connection to the database and caused some state to be lost. Workaround: I did a "systemctl restart box" and it picked up from where it left off.

                      Thanks for reporting! Let's see how common this is.

                      2021-06-25T12:07:36.791Z box:apps BoxError: connect ETIMEDOUT
                          at /home/yellowtent/box/src/settingsdb.js:26:36
                          at Query.queryCallback [as _callback] (/home/yellowtent/box/src/database.js:96:42)
                          at Query.Sequence.end (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24)
                          at /home/yellowtent/box/node_modules/mysql/lib/Pool.js:205:13
                          at Handshake.onConnect (/home/yellowtent/box/node_modules/mysql/lib/Pool.js:58:9)
                          at Handshake.<anonymous> (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:526:10)
                          at Handshake._callback (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:488:16)
                          at Handshake.Sequence.end (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24)
                          at Protocol.handleNetworkError (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:369:14)
                          at PoolConnection.Connection._handleNetworkError (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:418:18)
                      
                      1 Reply Last reply Reply Quote 1
                      • robi
                        robi last edited by robi

                        I have a similar issue where 55 apps were backed up, but then many had to reconfigure so that took 20m, and all of the apps redirect to the "you found a cloudron in the wild" page.

                        Restarted Nginx from services menu, no change.
                        Restarted box from CLI, no change. (It seems we could also use a restart button for the cloudron service.)
                        All my apps are offline.

                        Update: After 30m or so, apps are coming back online.

                        Life of Advanced Technology

                        1 Reply Last reply Reply Quote 1
                        • girish
                          girish Staff last edited by

                          The update has to do a "rebuild" of all the containers and services because of the multi-host changes. This can take a bit depending on how fast the server CPU and disk is. It's best to follow the progress under /home/yellowtent/platformdata/logs/box.log. That said 30m does seem quite excessive. Maybe the databases are taking a long time to come up? If you can see in the logs where the most time is spent, would be good to know.

                          d19dotca 1 Reply Last reply Reply Quote 0
                          • d19dotca
                            d19dotca @girish last edited by d19dotca

                            @girish Hi Girish, I think I had mentioned this a long time ago, but for what it's worth... I think it'd be helpful to include in the release notes a bit of an "info/alert" when there is to be more downtime than usual.

                            For me, it's alarming and jarring to see a random lengthy update when the usual update time to Cloudron is only a minute or two. So when it unexpectedly takes 10-30+ minutes (depending on performance of the server I guess) due to the containers needing to be re-created for example, it's not the best user experience.

                            I tend to make "30 minute" maintenance windows to sort of cover myself (and during off-peak hours of course) so the length of downtime isn't necessarily the main issue, but for me at least I was caught by surprise at the length it was taking and started to panic a little thinking something was wrong because it's so unusual for it to take that long. Thankfully I saw the containers being recreated when I looked at the system logs so knew it was okay and just needed some time, but I think having that pre-warning would be very helpful so admins at least know to expect a lengthier update instead of the usual update time.

                            --
                            Dustin Dauncey
                            www.d19.ca

                            1 Reply Last reply Reply Quote 3
                            • robi
                              robi last edited by

                              do these "rebuild" changes need to be a blocking operation?

                              can it be redone in a more gentle, async way and minimize overall downtime?

                              this will only amplify as we add more Cloudrons into the mix.

                              Life of Advanced Technology

                              1 Reply Last reply Reply Quote 0
                              • ?
                                A Former User last edited by A Former User

                                Having the same issue.
                                Screen Shot 2021-06-27 at 10.44.12 PM.png

                                Is this what others experienced?

                                Screen Shot 2021-06-27 at 10.47.16 PM.png

                                @girish This is my nginx logs... I don't have physical access to the server right now but I can drive over to take a look if you have ideas.

                                1 Reply Last reply Reply Quote 0
                                • ?
                                  A Former User last edited by A Former User

                                  Ok so now after a restart it looks like nginx just refuses to start...

                                  Fuck... guess im nuking my install, restoring from wasabi, and praying to the dieties i dont believe in that it just goes back to normal.

                                  @staff any idea what might have gone wrong? Can I set up a time with one of you to do a supervised update?

                                  Also I guess the contents of my volumes are fucked as well...

                                  girish 1 Reply Last reply Reply Quote 0
                                  • girish
                                    girish Staff @Guest last edited by

                                    @atridad That's not good, do you think I can have access to your server? If you enable SSH access, I can look into first thing tomorrow. thanks and sorry for the issue!

                                    ? 1 Reply Last reply Reply Quote 0
                                    • ?
                                      A Former User last edited by

                                      PXL_20210628_055812487~2.jpg

                                      This seems to be the error...

                                      1 Reply Last reply Reply Quote 0
                                      • ?
                                        A Former User @girish last edited by A Former User

                                        @girish Sure. I'll DM you

                                        EDIT: That works. Only thing is I cant access the dashboard which makes it hard to enable ssh access. I can make you an account if you need? I'm going to be up for the next 2 hours so, so let me know and I can set things up.

                                        M 1 Reply Last reply Reply Quote 0
                                        • M
                                          msbt App Dev @Guest last edited by

                                          @atridad you could run cloudron-support --enable-ssh - this should have the same effect

                                          ? 1 Reply Last reply Reply Quote 1
                                          • ?
                                            A Former User @msbt last edited by

                                            @msbt said in 6.3.3 a few quirks:

                                            cloudron-support --enable-ssh

                                            I love you.

                                            Seriously though thanks. I was pacing around trying to figure out how to do this lol

                                            1 Reply Last reply Reply Quote 0
                                            • nebulon
                                              nebulon Staff last edited by

                                              Ok at least for @atridad the issue was that for some reason old dashboard nginx configs were still around and referencing old SSL certificates which were now purged from the system. This made nginx fail to startup, causing the rest of the side-effects.

                                              necrevistonnezr 1 Reply Last reply Reply Quote 0
                                              • girish
                                                girish Staff last edited by

                                                Ah thanks, I had fixed this yesterday - https://git.cloudron.io/cloudron/box/-/commit/c052882de9c93ed00949514e234ade813a465598 . It will be in 6.3.4

                                                1 Reply Last reply Reply Quote 0
                                                • necrevistonnezr
                                                  necrevistonnezr @nebulon last edited by

                                                  @nebulon said in 6.3.3 a few quirks:

                                                  Ok at least for @atridad the issue was that for some reason old dashboard nginx configs were still around and referencing old SSL certificates which were now purged from the system. This made nginx fail to startup, causing the rest of the side-effects.

                                                  That had been my problem as well (more or less): https://forum.cloudron.io/topic/5263/cloudron-does-not-start-after-update-to-6-3-3

                                                  ? 1 Reply Last reply Reply Quote 1
                                                  • ?
                                                    A Former User @necrevistonnezr last edited by

                                                    @necrevistonnezr I just noticed your thread... whoops!

                                                    1 Reply Last reply Reply Quote 0
                                                    • d19dotca
                                                      d19dotca last edited by

                                                      @girish - I noticed 6.3.4 is out now... is this one that will again need more time than usual to update, or will it be a typical real quick one with nearly no downtime needed?

                                                      --
                                                      Dustin Dauncey
                                                      www.d19.ca

                                                      girish 2 Replies Last reply Reply Quote 0
                                                      • girish
                                                        girish Staff @d19dotca last edited by

                                                        @d19dotca 6.3.3 to 6.3.4 should be very quick. Only 6.2 -> 6.3 will take a bit to reconfigure containers.

                                                        1 Reply Last reply Reply Quote 0
                                                        • girish
                                                          girish Staff @d19dotca last edited by

                                                          @d19dotca Also, it's probably not clear from the UI because the UI is not optimized for the early upgraders. You can see https://git.cloudron.io/cloudron/box/-/blob/master/CHANGES for what exactly is the fix in the patch release.

                                                          1 Reply Last reply Reply Quote 0
                                                          • First post
                                                            Last post
                                                          Powered by NodeBB