SOLVED Are (encrypted) rsync backups not "incremental" on B2?

[necrevistonnezr]

I switched from a local backup + rclone to B2, using encrypted rsync backups.
Are those backups not incremental? It finished the first backup this morning with around 300 GB and I hit backup again an hour later - now it's at 21 % and added another 80 GB of backups...
Can that be right?

[girish]

@necrevistonnezr Does it say 'uploading' or 'copying' ? Also, how do you know it added 80GB of backups btw? Did you see that number in backblaze?

[necrevistonnezr]

@girish it said "copying" and I followed the increase of those numbers in Backblaze.

[robi]

@necrevistonnezr how are you using rclone?

[necrevistonnezr]

@robi said in Are (encrypted) rsync backups not "incremental" on B2?:

@necrevistonnezr how are you using rclone?

I used to (and might switch back to) backup to the file system via rsync and hardlinks and then the snapshot folder gets synced encrypted to Onedrive (where I have 1 TB laying around from an O365 subscription):

#!/bin/bash
d=$(date +%Y-%m-%d)
if pidof -o %PPID -x “$0”; then
echo “$(date “+%d.%m.%Y %T”) Exit, already running.”
exit 1
fi
rclone sync /media/CloudronBackup/snapshot odcrypt: -v --log-file=/media/"$d"_rclone-upload.log

whereas odcrypt is an encrypted version of the onedrive repository configured in rclone.

Has the added big bonus that I can just
rclone mount odcrypt /path/to/local/mount to access the unencrypted content of the backup

Has the further added bonus that the backup itself takes minutes, the upload then works in the background. Drawback: You have two copies of everything in you local filesystem.

Why only the snapshot folder? rclone does not do hardlinks. unfortunately. But maybe there are work arounds I haven't thought of or tried yet, e.g.: https://forum.rclone.org/t/rclone-hardlinked-incremental-backups-onedrive/17563

Ubuntu comes with an old version of rclone so install the newest! curl https://rclone.org/install.sh | sudo bash

[robi]

@necrevistonnezr oh that is great, and I want to set up something similar, more on that later.

Are you aware that tar has an incremental mode?

Then the local copy is just the tiny changes from the initial and the upload is also tiny.

Is odcrypt a feature of onedrive or rclone? What is the performance hit using it?

Have you been able to mount things like googledrive with the latest rclone?

I have a large gdrive and would love to point some mounted volumes there for Apps that store a large amount of (video) data.

[necrevistonnezr]

@robi said in Are (encrypted) rsync backups not "incremental" on B2?:

@necrevistonnezr oh that is great, and I want to set up something similar, more on that later.

Are you aware that tar has an incremental mode?
Then the local copy is just the tiny changes from the initial and the upload is also tiny.

No, is that built-in into Cloudron's backup?

Is odcrypt a feature of onedrive or rclone? What is the performance hit using it?

odcrypt is just a random name I gave it. You first configure onedrive or google drive or whatever (https://rclone.org/docs/) and then configure encrpytion (https://rclone.org/crypt/). I haven't seen (albeit not checked, either) a performance hit.

Have you been able to mount things like googledrive with the latest rclone?

I can mount Onedrive encrypted without problem, so Google Drive should work as it's supported (https://rclone.org/commands/rclone_mount/)

[girish]

@necrevistonnezr Ah ok, so it is sort of working as the code is designed. Currently, rsync is optimizing on uploads and only uploading incremental changes. But the final storage is not incremental, every backup (i.e timestamped directory in the remote) is a full complete standalone backup.

[robi]

@necrevistonnezr said

No, is that built-in into Cloudron's backup?

No, but if you test it and provide results, it may get added.

I can mount Onedrive encrypted without problem, so Google Drive should work as it's supported (https://rclone.org/commands/rclone_mount/)

Perfect, I have to get to this soon, thanks for the detail!

[necrevistonnezr]

@girish said in Are (encrypted) rsync backups not "incremental" on B2?:

@necrevistonnezr Ah ok, so it is sort of working as the code is designed. Currently, rsync is optimizing on uploads and only uploading incremental changes. But the final storage is not incremental, every backup (i.e timestamped directory in the remote) is a full complete standalone backup.

So that would result in huge amounts of space on the remote backup facility if you choose a certain retention, in my case (3 daily, 4 weekly, 6 monthly) something like 15 x 300 GB = 4,5 TB(!), correct?.

It would be great if there was some deduplication done beforehand...? Or is using the --copy-destoption? (see some discussion here: https://www.linuxquestions.org/questions/linux-server-73/incremental-backups-with-rsync-link-dest-4175603276/)

[necrevistonnezr]

@robi BTW restic works with rclone, meaning you can use restic on anything that rclone can access (which is quite a lot, though restic already works with many cloud providers out of the box...

[robi]

@necrevistonnezr oh sweet, wasn't familiar with restic. And sure makes sense, if rclone can mount it, restic can backup to it.

It sounds like @staff should take a look at restic+rclone for greatly enhancing backups in Cloudron with minimal work!

We'd also get double free dedupe (once with restic and once on the target side with rclone) and minimal incremental backups, so efficient!

[girish]

@necrevistonnezr said in Are (encrypted) rsync backups not "incremental" on B2?:

So that would result in huge amounts of space on the remote backup facility if you choose a certain retention, in my case (3 daily, 4 weekly, 6 monthly) something like 15 x 300 GB = 4,5 TB(!), correct?.

That's correct. There is no de-duplication on the backend.

[necrevistonnezr]

@girish
Thanks! You should seriously check out restic.

• The daily incremental backups I set up consume 200 MB if I changed 200 MBs over a day on a 300 GB repository.
• Takes 20 Minutes.
• Backups are encrypted.
• Many providers built in, the rest works via rclone. Backups are mountable for easy access to individual files.

[marcusquinn]

@necrevistonnezr said in Are (encrypted) rsync backups not "incremental" on B2?:

restic

Interesting, their sponsors platform looks interesting too: https://appscode.com/

[girish]

For restic, can you upvote here - https://forum.cloudron.io/topic/1575/backup-improvements-restic-backend ? It should be possible to add a new restic backend like tgz/rsync. Though, we still have to figure out how we can "support" it. But that's a separate topic.