Cloudron+ZFS?
-
@necrevistonnezr beautiful
Funny how you have to prune cuz it forgets.
-
@robi
I useTools
- rclone: https://rclone.org/docs/
- restic: https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#other-services-via-rclone
- ssmtp: https://wiki.archlinux.org/title/SSMTP
Installation
- Install tools above via apt
- afterwards update to latest version (repo versions are old):
sudo restic self-update && sudo rclone selfupdate
Setup rclone
- Enter an interactive setup process via
rclone config - in my case I use Onedrive as it has 1TB of space coming with my Office 365 subscription
- for the rest of this summary, we assume you gave it the repository name "REPOSITORY"
- details at https://rclone.org/commands/rclone_config/
Setup restic
- set up a backup repository
restic -r rclone:REPOSITORY init - for a subfolder on onedrive just use
restic -r rclone:REPOSITORY:subfolder init - save password that you gave the repository in file
/home/USER/resticpw - details at https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#other-services-via-rclone
Setup SSMTP
- for receiving backup results, otherwise not needed
- See https://wiki.archlinux.org/title/SSMTP
Cloudron Backup settings
- Provider: mountpoint
- Location:
/media/CloudronBackup(<-- obviously adjust to your settings) - this creates a snapshot at
/media/CloudronBackup/snapshotfor the current backup - Storage Format: rsync
- Adjust schedule and retention to your liking
Backup, Prune and Check scripts
restic-cron-backup.sh
#!/bin/bash d=$(date +%Y-%m-%d) if pidof -o %PPID -x “$0”; then echo “$(date “+%d.%m.%Y %T”) Exit, already running.” exit 1 fi restic -r rclone:REPOSITORY:subfolder backup /media/CloudronBackup/snapshot -p=/home/USER/resticpw restic -r rclone:REPOSITORY:subfolder forget --keep-monthly 12 --keep-weekly 5 --keep-daily 14 -p=/home/USER/resticpw restic -r rclone:REPOSITORY:subfolder check --read-data-subset=2% -p=/home/USER/resticpw exitFirst line does the backup (incremental, encrypted), second line is the backup retention, third line checks a random 2 % of all data for errors
restic-cron-prune.sh
#!/bin/bash d=$(date +%Y-%m-%d) if pidof -o %PPID -x “$0”; then echo “$(date “+%d.%m.%Y %T”) Exit, already running.” exit 1 fi restic -r rclone:REPOSITORY:subfolder prune -p=/home/USER/resticpw exitremoves unused data from the repository, I run this once a week
restic-cron-check.sh
#!/bin/bash d=$(date +%Y-%m-%d) if pidof -o %PPID -x “$0”; then echo “$(date “+%d.%m.%Y %T”) Exit, already running.” exit 1 fi restic -r rclone:REPOSITORY:subfolder check --read-data -p=/home/USER/resticpw exitchecks all data for errors, I run this once a week
Crontab
30 2 * * * sh /home/USER/restic-cron-backup.sh | mailx -s "Restic Backup Results" server@mydomain.com 1 5 1 * * sh /home/USER/restic-cron-prune.sh | mailx -s "Restic Prune Results" server@mydomain.com 1 8 1 * * sh /home/USER/restic-cron-check.sh | mailx -s "Restic Full Check Results" server@mydomain.comBackup daily at 2:30, prune and check once a week. Receive results to specified mail
Mount backups
Just to be complete: You can mount restic backups locally like
restic -r rclone:REPOSITORY:subfolder mount /media/resticmount/ -p=/home/USER/resticpw && cd /media/resticmount
obviously adjust/media/resticmount/to your settings; allows you to browse and copy from full snapshots for each backup@necrevistonnezr amazing! Thank you for sharing
I am pondering a similar approach and currently back up to a secondary internal drive via rsync without encryption, although the backup drive is using LUKS.
I wonder if using rclone crypt instead of encrypting the files via restic would gain any advantages (e.g. maximum file / folder names, depth, speed)?
Cloudron > rsync to local drive > rclone/crypt via restic > remote
👉 Find our more www.bebraver.online
-
@necrevistonnezr amazing! Thank you for sharing
I am pondering a similar approach and currently back up to a secondary internal drive via rsync without encryption, although the backup drive is using LUKS.
I wonder if using rclone crypt instead of encrypting the files via restic would gain any advantages (e.g. maximum file / folder names, depth, speed)?
Cloudron > rsync to local drive > rclone/crypt via restic > remote
Just reading up on restic and encryption, etc and may just skip the rclone part as I am looking to either go to Wasabi or Hetzner Storagebox.
However, I kinda like the crypt part and am looking for any comparisons between rclone/crypt with restic and restic w/ encryption in terms of time it takes to backup and any drawbacks.
👉 Find our more www.bebraver.online
-
Just reading up on restic and encryption, etc and may just skip the rclone part as I am looking to either go to Wasabi or Hetzner Storagebox.
However, I kinda like the crypt part and am looking for any comparisons between rclone/crypt with restic and restic w/ encryption in terms of time it takes to backup and any drawbacks.
@3246 said in Cloudron+ZFS?:
Just reading up on restic and encryption, etc and may just skip the rclone part as I am looking to either go to Wasabi or Hetzner Storagebox.
However, I kinda like the crypt part and am looking for any comparisons between rclone/crypt with restic and restic w/ encryption in terms of time it takes to backup and any drawbacks.
Just to be clear: The encryption (as well as deduplication, the repository, data integrity checks, etc.) is completely handled by restic. rclone is just the "transporter tool" that copies data to providers that the restic does not handle (restic out-of-the-box currently handles SFTP, REST-Server, Minio, Wasabi, etc. see https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html).
-
@3246 said in Cloudron+ZFS?:
Just reading up on restic and encryption, etc and may just skip the rclone part as I am looking to either go to Wasabi or Hetzner Storagebox.
However, I kinda like the crypt part and am looking for any comparisons between rclone/crypt with restic and restic w/ encryption in terms of time it takes to backup and any drawbacks.
Just to be clear: The encryption (as well as deduplication, the repository, data integrity checks, etc.) is completely handled by restic. rclone is just the "transporter tool" that copies data to providers that the restic does not handle (restic out-of-the-box currently handles SFTP, REST-Server, Minio, Wasabi, etc. see https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html).
-
@necrevistonnezr thank you. Is the encryption always part of restic or optional? It looks like it's baked-in the way repos are build, right?
@3246 Yes. There is discussions about it (see https://github.com/restic/restic/issues/1018) but that's actually on of the features of restic: It's encryption done right.
And being able to mount the repositories and directly access them is just fantastic. -
@3246 Yes. There is discussions about it (see https://github.com/restic/restic/issues/1018) but that's actually on of the features of restic: It's encryption done right.
And being able to mount the repositories and directly access them is just fantastic.@necrevistonnezr very interesting. Thank you for sharing that link too. I'll give it a try without the rclone step soon
-
I cribbed off the scripts kindly provided by @necrevistonnezr and am using restic straight to a Hetzner Storagebox.
The first upload is running:
[23:53] 0.59% 427 files 4.571 GiB, total 1344856 files 772.976 GiB, 0 errors ETA 66:59:14Not quite sure where the 772GB are coming from as the directory is much smaller?
390G /mnt/local_backups/rsync/2022-05-03-130134-941 1.4G /mnt/local_backups/rsync/snapshot -
All these posts about backing up locally and then to somewhere else are really useful, but a bit hidden away in this thread about something pretty tenuously related.
Methinks @staff should move some of them into a new thread of their own
-
I cribbed off the scripts kindly provided by @necrevistonnezr and am using restic straight to a Hetzner Storagebox.
The first upload is running:
[23:53] 0.59% 427 files 4.571 GiB, total 1344856 files 772.976 GiB, 0 errors ETA 66:59:14Not quite sure where the 772GB are coming from as the directory is much smaller?
390G /mnt/local_backups/rsync/2022-05-03-130134-941 1.4G /mnt/local_backups/rsync/snapshot@3246 Are you backing up
/snapshot/or the parent directory?
/snapshot/is sufficient for a daily backup as it holds the current status of all files - versioning etc. is done by restic.
Also, how did you calculate these dirsizes?
