Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    Solved Backup failure retry wait times should be shortened from 4 hours

    Feature Requests
    backups
    4
    11
    199
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • d19dotca
      d19dotca last edited by girish

      I've noticed that backup failures trigger a retry 4 hours later. This "4 hours" retry wait time isn't customizable from the looks of it, and (maybe it's just me that has this opinion but) it seems to be quite a long time to wait for a retry.

      I'd like to request either of the following be changed or added to Cloudron:

      • Customizable option for retry wait time
      • Shorten it to somewhere between 30 minutes and 2 hours at most

      Unless space is an issue and there maybe needs to be some additional space added which can maybe take some time before an admin can get to it, I can't quite think of a reason why it needs to wait 4 hours for the next one.

      This thought has always been on my mind, but it's more apparent to me now that 5.5.0 allows us to customize the backup schedule. In my example, it's backing up every 6 hours (4 times a day total), and so retrying the backup 4 hours afterwards means there's then just a 2 hour gap between that and the next one (if the retry succeeds), which seems a little bit pointless to have auto backups just two hours apart.

      Hopefully the above makes sense, but please let me know if I can clarify at all. 🙂

      --
      Dustin Dauncey
      www.d19.ca

      girish 1 Reply Last reply Reply Quote 0
      • girish
        girish Staff @d19dotca last edited by girish

        @d19dotca in 5.5, we have removed the retry logic entirely. If the backup fails , it gives a notification and only backs up next time based on the schedule. The retry logic doesn't work well because then it might conflict with work hours again and it was the whole reason to make the backup schedule configurable in the first place. I think for general network errors there is already separate retry logic in place independent of this. For any other convoluted errors like mount point not found, the retry wont help.

        d19dotca 1 Reply Last reply Reply Quote 0
        • d19dotca
          d19dotca @girish last edited by d19dotca

          @girish That's good to hear, but if that's true then I think there may be just a small defect because the notification I receive in Cloudron when it fails still has the line about retrying in 4 hours. Not the email, but the actual notification inside of Cloudron.

          --
          Dustin Dauncey
          www.d19.ca

          1 Reply Last reply Reply Quote 1
          • girish
            girish Staff last edited by

            @d19dotca Ah, you are right. I forgot to fix that!

            There are some more changes to the backup system (just putting it here, since it's not obvious):

            • The backup is now run with a nice of 15. This makes sure that it gets low priority if the Cloudron is doing other things.
            • It's run with a configurable memory limit. This memory limit is in Advanced -> Settings. This is useful if you want to do faster upload (by increasing concurrency values). This necessarily means you have to give the task more memory.
            • There is currently a timeout of 12 hours for the task. If people are hitting this limit, I will bump this up.
            d19dotca imc67 2 Replies Last reply Reply Quote 1
            • d19dotca
              d19dotca @girish last edited by d19dotca

              @girish This needs to be added to the docs too on the concurrency limits and such. Just went to reference it to set appropriate values and didn’t find it. For what it’s worth, I tried with the defaults (10 concurrency) on all three of those settings, and eventually bumped it to 100. But I didn’t notice any real difference. I guess those settings apply more to remote systems instead of ext4 mounted disks?

              --
              Dustin Dauncey
              www.d19.ca

              1 Reply Last reply Reply Quote 0
              • imc67
                imc67 translator @girish last edited by

                @girish said in Backup failure retry wait times should be shortened from 4 hours:

                by increasing concurrency values

                Where can I find these settings? I only see the memory limit setting (using S3)

                1 Reply Last reply Reply Quote 0
                • nebulon
                  nebulon Staff last edited by

                  The upload, download, copy concurrency settings only apply to the rsync strategy, so they are only shown below the memory limit if rsync is used.

                  1 Reply Last reply Reply Quote 1
                  • girish
                    girish Staff last edited by

                    @d19dotca Right, concurrency settings only apply to s3 backends that use rsync. I have fixed the ui.

                    1 Reply Last reply Reply Quote 1
                    • imc67
                      imc67 translator last edited by

                      The backup changes worked out very well for us!!!

                      Cloudron #1 (5.4.1) was 35 minutes for about 14GB (tgz) and now (5.5) 3 minutes with highest concurrency limits / memory in rsync to external Minio.

                      Cloudron #2 (5.4.1) was about 1 hour for about 38GB (tgz) and now (5.5) 36 minutes with highest concurrency limits / memory in rsync to external Minio.

                      Thanks @girish for this wonderfull and welcome update (also for being able to set own times and intervals)!

                      1 Reply Last reply Reply Quote 0
                      • girish
                        girish Staff last edited by

                        @imc67 Thanks! That's the sort of improvement I expected to see with the new concurrency settings.

                        1 Reply Last reply Reply Quote 0
                        • girish
                          girish Staff last edited by girish

                          @d19dotca I have added some docs now.

                          https://cloudron.io/documentation/backups/#schedule has info on the timeout and nice.

                          https://cloudron.io/documentation/backups/#concurrency-settings on the concurrency settings.

                          1 Reply Last reply Reply Quote 1
                          • First post
                            Last post
                          Powered by NodeBB