Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    Solved update to cloudron 5.4 crashes complete cloudron server

    Support
    update crash
    4
    10
    300
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • imc67
      imc67 translator last edited by girish

      Just updated (or trying to) to 5.4 and the whole Cloudron server crashed, all apps down, dashboard down. CPU busy, no network activity.

      Via chat @nebulon and @girish were not available and I coincidently had a server snapshot of 2 days old so could restore on server level.

      Now I'm missing 2 days data and worse 2 days email!

      How to quickly restore (because new mails are coming in) at least email from the last backup?

      1 Reply Last reply Reply Quote 0
      • nebulon
        nebulon Staff last edited by

        Hi,

        what is the actual crash so we can fix it? Like something in the logs?

        Regarding the restore, not sure I understand this correctly, so you have restored a server snapshot from 2 days ago already or plan to do so?

        If the latter, please enable remote ssh support via cloudron-support --enable-ssh and send a mail to support@cloudron.io with your domain. Then we can debug and fix this.

        imc67 1 Reply Last reply Reply Quote 0
        • imc67
          imc67 translator @nebulon last edited by

          @nebulon sorry but in the meanwhile I was busy for hours to get everything back online, this is what I did and found out:

          • restored server snapshot (-2 days) as that was for me at that time the only thing I could do

          • was looking for a way to restore a backup of box (because of email) while running the -2 days snapshot but in the docs there is nothing mentioned about restoring a box

          • 'rented' a new NetCup VPS, started it with the Cloudron image ... but restoring a backup was not possible because that image is already on 5.4

          • I reinstalled the VPS with Ubuntu and manually installed Cloudron 5.3.4

          • then I was able to restore the backup made just before the 5.4 upgrade

          Lessons learned: always make a snapshot before a Cloudron update

          Question: how can one restore a box (or specifically email) on a running Cloudron? A few years ago a user (on our previous host) accidentally deleted all his folders in his emailbox, then I was able (DirectAdmin) to only restore his box (it had an hourly backup system)?

          1 Reply Last reply Reply Quote 1
          • nebulon
            nebulon Staff last edited by

            Do you happen to remember what was maybe crashing, so we can investigate what the root issue was?

            Restoring the main Cloudron system database,mailboxes,... (ie the "box") is not individually possible, as apps depend on that and a rollback of that while the apps already using newer state could result in inconsistencies.

            Generally I think what you did in the end was the correct approach to restore the whole server on a new VPS using the backup made prior to the update. We may have to improve that flow a bit further, since in such situations the stress level is already high, so it should be a smooth restore path.

            imc67 1 Reply Last reply Reply Quote 2
            • imc67
              imc67 translator @nebulon last edited by

              @nebulon I SSH'd into the server during the 'crash' and did a 'top' command, I remember seeing only some processes using load something like 'Docker' and 'Node'.

              I tried a reboot but after that I wasn't even able to SSH into the VPS anymore, that was the moment I thought of rebuilding/moving.

              Herewith a screenshot from my Zabbix dashboard, backup started around 7:10h finished just after 8:05h and then the update started and "crashed".

              Schermafbeelding 2020-07-18 om 13.04.04.png

              1 Reply Last reply Reply Quote 1
              • nebulon
                nebulon Staff last edited by

                Unfortunately that does not give us much information to debug this further. Next time if that happens, it is worth looking into the log files at /home/yellowtent/platformdata/logs/ most likely the issue will surface within the logs.

                1 Reply Last reply Reply Quote 0
                • imc67
                  imc67 translator last edited by

                  Just the latest update: the fresh new VPS with Cloudron Pro restore automatically updated last night to 5.4 without any issue. Also my two other Cloudron Pro instances had an automatic update and went well.

                  Next time (hopefully not needed) I will download logs before a restore to make debug possible 😉.

                  d19dotca 1 Reply Last reply Reply Quote 0
                  • d19dotca
                    d19dotca @imc67 last edited by

                    @imc67 @girish Are pre-release versions auto-updating? I thought they had to be manually installed? I ask because now I'm worried mine may auto-update to a pre-release version on a production system.

                    --
                    Dustin Dauncey
                    www.d19.ca

                    1 Reply Last reply Reply Quote 0
                    • nebulon
                      nebulon Staff last edited by

                      There should currently be no automatic update for 5.4 however if manually checked for updates, then the update can be applied already.

                      1 Reply Last reply Reply Quote 1
                      • girish
                        girish Staff last edited by

                        @imc67 I made https://git.cloudron.io/cloudron/box/-/issues/717 for keeping mail backups separately from the box backups.

                        1 Reply Last reply Reply Quote 4
                        • First post
                          Last post
                        Powered by NodeBB