Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    Solved Are (encrypted) rsync backups not "incremental" on B2?

    Support
    backblaze rsync backups
    4
    16
    494
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • necrevistonnezr
      necrevistonnezr last edited by girish

      I switched from a local backup + rclone to B2, using encrypted rsync backups.
      Are those backups not incremental? It finished the first backup this morning with around 300 GB and I hit backup again an hour later - now it's at 21 % and added another 80 GB of backups...
      Can that be right?

      girish robi 2 Replies Last reply Reply Quote 0
      • girish
        girish Staff @necrevistonnezr last edited by

        @necrevistonnezr Does it say 'uploading' or 'copying' ? Also, how do you know it added 80GB of backups btw? Did you see that number in backblaze?

        necrevistonnezr 1 Reply Last reply Reply Quote 0
        • necrevistonnezr
          necrevistonnezr @girish last edited by

          @girish it said "copying" and I followed the increase of those numbers in Backblaze.

          girish 1 Reply Last reply Reply Quote 0
          • robi
            robi @necrevistonnezr last edited by

            @necrevistonnezr how are you using rclone?

            Life of Gratitude.
            Life of Advanced Technology

            necrevistonnezr 1 Reply Last reply Reply Quote 0
            • necrevistonnezr
              necrevistonnezr @robi last edited by necrevistonnezr

              @robi said in Are (encrypted) rsync backups not "incremental" on B2?:

              @necrevistonnezr how are you using rclone?

              I used to (and might switch back to) backup to the file system via rsync and hardlinks and then the snapshot folder gets synced encrypted to Onedrive (where I have 1 TB laying around from an O365 subscription):

              #!/bin/bash
              d=$(date +%Y-%m-%d)
              if pidof -o %PPID -x “$0”; then
              echo “$(date “+%d.%m.%Y %T”) Exit, already running.”
              exit 1
              fi
              rclone sync /media/CloudronBackup/snapshot odcrypt: -v --log-file=/media/"$d"_rclone-upload.log
              

              whereas odcrypt is an encrypted version of the onedrive repository configured in rclone.

              Has the added big bonus that I can just
              rclone mount odcrypt /path/to/local/mount to access the unencrypted content of the backup

              Has the further added bonus that the backup itself takes minutes, the upload then works in the background. Drawback: You have two copies of everything in you local filesystem.

              Why only the snapshot folder? rclone does not do hardlinks. unfortunately. But maybe there are work arounds I haven't thought of or tried yet, e.g.: https://forum.rclone.org/t/rclone-hardlinked-incremental-backups-onedrive/17563

              Ubuntu comes with an old version of rclone so install the newest! curl https://rclone.org/install.sh | sudo bash

              robi 1 Reply Last reply Reply Quote 0
              • robi
                robi @necrevistonnezr last edited by

                @necrevistonnezr oh that is great, and I want to set up something similar, more on that later.

                Are you aware that tar has an incremental mode?

                Then the local copy is just the tiny changes from the initial and the upload is also tiny.

                Is odcrypt a feature of onedrive or rclone? What is the performance hit using it?

                Have you been able to mount things like googledrive with the latest rclone?

                I have a large gdrive and would love to point some mounted volumes there for Apps that store a large amount of (video) data.

                Life of Gratitude.
                Life of Advanced Technology

                necrevistonnezr 1 Reply Last reply Reply Quote 0
                • necrevistonnezr
                  necrevistonnezr @robi last edited by

                  @robi said in Are (encrypted) rsync backups not "incremental" on B2?:

                  @necrevistonnezr oh that is great, and I want to set up something similar, more on that later.

                  Are you aware that tar has an incremental mode?
                  Then the local copy is just the tiny changes from the initial and the upload is also tiny.

                  No, is that built-in into Cloudron's backup?

                  Is odcrypt a feature of onedrive or rclone? What is the performance hit using it?

                  odcrypt is just a random name I gave it. You first configure onedrive or google drive or whatever (https://rclone.org/docs/) and then configure encrpytion (https://rclone.org/crypt/). I haven't seen (albeit not checked, either) a performance hit.

                  Have you been able to mount things like googledrive with the latest rclone?

                  I can mount Onedrive encrypted without problem, so Google Drive should work as it's supported (https://rclone.org/commands/rclone_mount/)

                  robi 1 Reply Last reply Reply Quote 0
                  • girish
                    girish Staff @necrevistonnezr last edited by

                    @necrevistonnezr Ah ok, so it is sort of working as the code is designed. Currently, rsync is optimizing on uploads and only uploading incremental changes. But the final storage is not incremental, every backup (i.e timestamped directory in the remote) is a full complete standalone backup.

                    necrevistonnezr 1 Reply Last reply Reply Quote 0
                    • robi
                      robi @necrevistonnezr last edited by

                      @necrevistonnezr said

                      No, is that built-in into Cloudron's backup?

                      No, but if you test it and provide results, it may get added.

                      I can mount Onedrive encrypted without problem, so Google Drive should work as it's supported (https://rclone.org/commands/rclone_mount/)

                      Perfect, I have to get to this soon, thanks for the detail!

                      Life of Gratitude.
                      Life of Advanced Technology

                      necrevistonnezr 1 Reply Last reply Reply Quote 0
                      • necrevistonnezr
                        necrevistonnezr @girish last edited by necrevistonnezr

                        @girish said in Are (encrypted) rsync backups not "incremental" on B2?:

                        @necrevistonnezr Ah ok, so it is sort of working as the code is designed. Currently, rsync is optimizing on uploads and only uploading incremental changes. But the final storage is not incremental, every backup (i.e timestamped directory in the remote) is a full complete standalone backup.

                        So that would result in huge amounts of space on the remote backup facility if you choose a certain retention, in my case (3 daily, 4 weekly, 6 monthly) something like 15 x 300 GB = 4,5 TB(!), correct?.

                        It would be great if there was some deduplication done beforehand...? Or is using the --copy-destoption? (see some discussion here: https://www.linuxquestions.org/questions/linux-server-73/incremental-backups-with-rsync-link-dest-4175603276/)

                        girish 1 Reply Last reply Reply Quote 1
                        • necrevistonnezr
                          necrevistonnezr @robi last edited by

                          @robi BTW restic works with rclone, meaning you can use restic on anything that rclone can access (which is quite a lot, though restic already works with many cloud providers out of the box...

                          robi 1 Reply Last reply Reply Quote 0
                          • robi
                            robi @necrevistonnezr last edited by

                            @necrevistonnezr oh sweet, wasn't familiar with restic. And sure makes sense, if rclone can mount it, restic can backup to it.

                            It sounds like @staff should take a look at restic+rclone for greatly enhancing backups in Cloudron with minimal work!

                            We'd also get double free dedupe (once with restic and once on the target side with rclone) and minimal incremental backups, so efficient! ❤

                            Life of Gratitude.
                            Life of Advanced Technology

                            1 Reply Last reply Reply Quote 1
                            • girish
                              girish Staff @necrevistonnezr last edited by

                              @necrevistonnezr said in Are (encrypted) rsync backups not "incremental" on B2?:

                              So that would result in huge amounts of space on the remote backup facility if you choose a certain retention, in my case (3 daily, 4 weekly, 6 monthly) something like 15 x 300 GB = 4,5 TB(!), correct?.

                              That's correct. There is no de-duplication on the backend.

                              necrevistonnezr 1 Reply Last reply Reply Quote 0
                              • necrevistonnezr
                                necrevistonnezr @girish last edited by

                                @girish
                                Thanks! You should seriously check out restic.

                                • The daily incremental backups I set up consume 200 MB if I changed 200 MBs over a day on a 300 GB repository.
                                • Takes 20 Minutes.
                                • Backups are encrypted.
                                • Many providers built in, the rest works via rclone. Backups are mountable for easy access to individual files.
                                marcusquinn 1 Reply Last reply Reply Quote 1
                                • marcusquinn
                                  marcusquinn @necrevistonnezr last edited by

                                  @necrevistonnezr said in Are (encrypted) rsync backups not "incremental" on B2?:

                                  restic

                                  Interesting, their sponsors platform looks interesting too: https://appscode.com/

                                  We're not here for a long time - but we are here for a good time :)
                                  Jersey/UK
                                  Work & Ecommerce Advice: https://brandlight.org
                                  Personal & Software Tips: https://marcusquinn.com

                                  1 Reply Last reply Reply Quote 0
                                  • girish
                                    girish Staff last edited by

                                    For restic, can you upvote here - https://forum.cloudron.io/topic/1575/backup-improvements-restic-backend ? It should be possible to add a new restic backend like tgz/rsync. Though, we still have to figure out how we can "support" it. But that's a separate topic.

                                    1 Reply Last reply Reply Quote 1
                                    • First post
                                      Last post
                                    Powered by NodeBB