Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Navigation

    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    Improve Clone/Backup/Restore Speed

    Feature Requests
    backups clone
    9
    24
    182
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • robi
      robi last edited by robi

      Looking at the progress of a clone from a backup on a remote S3 share, it seems to restore individual files, one by one, which takes hours for larger apps.

      Can the operation be restructured to quickly pull the full backup locally and then copy the files to the container much faster?

      d19dotca 1 Reply Last reply Reply Quote 1
      • d19dotca
        d19dotca @robi last edited by

        @robi This likely has to do with the remote-nature of your backups. Using a mounted disk allows cloning to be super quick. I know there was room for improvement before and changes later made to allow for faster backups, but not certain if that impacts the restores too.

        1 Reply Last reply Reply Quote 0
        • girish
          girish Staff last edited by

          Is this rsync or tgz? You can now configure the Download concurrency in Advanced Settings.

          robi 1 Reply Last reply Reply Quote 0
          • robi
            robi @girish last edited by

            @girish currently it's set to rsync and default concurrency of 10.

            Increasing concurrency isn't as useful when all the files are fetched one by one. It's too many requests.

            Object storage should NOT be treated like a filesystem because it isn't one.

            The large object should be requested in its entirety.

            This is something I worked on at IBM as few people on the planet understood this concept.

            As for a clone, it would be much faster now to install a new copy of the same app, and apply the few deltas. But that's not what's happening.

            How can we speed things up?

            1 Reply Last reply Reply Quote 0
            • girish
              girish Staff last edited by

              @robi said in Improve Clone Speed:

              The large object should be requested in its entirety.

              Mmm, I am not following. What do you mean by large object? If the files are stored separately as rsync does, they have to be fetched separately as well.

              As for a clone, it would be much faster now to install a new copy of the same app, and apply the few deltas. But that's not what's happening.

              That's honestly very complex (might even be impossible) since you have to delta databases, caches etc.

              robi 1 Reply Last reply Reply Quote 1
              • robi
                robi @girish last edited by

                @girish said in Improve Clone Speed:

                Mmm, I am not following. What do you mean by large object? If the files are stored separately as rsync does, they have to be fetched separately as well.

                Then I am definitely using the wrong method.

                As for a clone, it would be much faster now to install a new copy of the same app, and apply the few deltas. But that's not what's happening.

                That's honestly very complex (might even be impossible) since you have to delta databases, caches etc.

                For a clone, it would be simpler to duplicate the database/caches, etc over the default install and not compare anything.

                1 Reply Last reply Reply Quote 0
                • girish
                  girish Staff last edited by

                  @robi said in Improve Clone Speed:

                  For a clone, it would be simpler to duplicate the database/caches, etc over the default install and not compare anything.

                  Yes, good point 🙂 And that's probably the solution to make things faster.. Currently, our clone system is tied to a backup. So, this feature request is "clone a live app" (i.e skip this backup stuff altogether).

                  1 Reply Last reply Reply Quote 3
                  • robi
                    robi last edited by

                    @girish are you aware of the incremental feature of tar ?

                    This feature is provided by tar via an argument -listed-incremental=snapshot-file where a "snapshot-file" is a special file maintained by the tar command to determine the files that are been added,modified or deleted.

                    That will speed things up.

                    yusf 2 Replies Last reply Reply Quote 3
                    • yusf
                      yusf @robi last edited by yusf

                      @robi Would this mean that we can have incremental, encrypted tar backups? That would be amazing as it would solve all my backup woes by combining the best of both methods.

                      robi 1 Reply Last reply Reply Quote 2
                      • robi
                        robi @yusf last edited by

                        @yusf that's probably a better question for @girish but likely yes.

                        jdaviescoates 1 Reply Last reply Reply Quote 1
                        • jdaviescoates
                          jdaviescoates @robi last edited by jdaviescoates

                          @robi @yusf @girish

                          More info here:
                          https://www.gnu.org/software/tar/manual/html_node/Incremental-Dumps.html

                          And here:
                          https://serverfault.com/a/569668

                          But perhaps what we really want is Borg?
                          https://borgbackup.readthedocs.io/en/stable/

                          It's recommended in that Server Fault post, but also when I asked Hetzner about backups I can download (you can't download backup from their native backup solution) to store off-Hetzer, they suggested I use it, here is their guide:

                          https://community.hetzner.com/tutorials/install-and-configure-borgbackup

                          But that's ^ all a bit too technical for me. I use Cloudron so I don't have to bother with all that. So I'd love to just have Borg as a backup option on Cloudron 🙂

                          1 Reply Last reply Reply Quote 0
                          • Lonk
                            Lonk last edited by

                            Incremental backups. I thought we had those already, but if we don't - we should vote to support them.

                            jdaviescoates 1 Reply Last reply Reply Quote 0
                            • jdaviescoates
                              jdaviescoates @Lonk last edited by

                              @Lonk said in Improve Clone/Backup/Restore Speed:

                              Incremental backups. I thought we had those already, but if we don't - we should vote to support them.

                              We do have them with the rsync option.

                              Lonk 1 Reply Last reply Reply Quote 0
                              • necrevistonnezr
                                necrevistonnezr last edited by necrevistonnezr

                                Actually, rsync and rclone-ing the incremental backups - encrypted - to Onedrive has been very reliable for me the last years. rclone is such a fantastic tool.

                                1 Reply Last reply Reply Quote 1
                                • Lonk
                                  Lonk @jdaviescoates last edited by

                                  @jdaviescoates I'm using .tgz with Backblaze right now since tgz was the default I didn't look into it. Should I switch to rsync for the benefit of incremental changes or are there cons like @robi is trying to solve (his cloning / restoring speed suggestions).

                                  Is "Incremental TAR files" the best of both worlds, basically?

                                  marcusquinn 1 Reply Last reply Reply Quote 0
                                  • marcusquinn
                                    marcusquinn @Lonk last edited by

                                    @Lonk If you use rsync, use Wasabi as it has no ingress costs. Also, in Backblaze, check the lifecycle settings on all buckets to make sure you're not paying to insure infinite versions of versions, just change the setting for each in there to just store the latest.

                                    Lonk 1 Reply Last reply Reply Quote 0
                                    • Lonk
                                      Lonk @marcusquinn last edited by

                                      @marcusquinn said in Improve Clone/Backup/Restore Speed:

                                      @Lonk If you use rsync, use Wasabi as it has no ingress costs. Also, in Backblaze, check the lifecycle settings on all buckets to make sure you're not paying to insure infinite versions of versions, just change the setting for each in there to just store the latest.

                                      I did have infinite versions on, thanks for saving me there, I owe ya!

                                      marcusquinn 1 Reply Last reply Reply Quote 1
                                      • marcusquinn
                                        marcusquinn @Lonk last edited by

                                        @Lonk Everyone does as the sneaky f***ers make it the default. Must have lost thousands of dollars before I found that where everyone else missed it.

                                        marcusquinn 1 Reply Last reply Reply Quote 0
                                        • marcusquinn
                                          marcusquinn @marcusquinn last edited by

                                          Wasabi don't make it the default though, and have a much better interface. I'm dropping Backblaze from my recommendations for S3 needs and only use if for personal machine backups which don't have all those extra costs.

                                          1 Reply Last reply Reply Quote 0
                                          • yusf
                                            yusf @robi last edited by

                                            @robi said in Improve Clone/Backup/Restore Speed:

                                            This feature is provided by tar via an argument -listed-incremental=snapshot-file where a "snapshot-file" is a special file maintained by the tar command to determine the files that are been added,modified or deleted.

                                            Have @girish or @nebulon seen this? 😃

                                            1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post