Tutorial: remote backup of local Cloudron backup snapshots with restic / rclone
-
@timconsidine said in Tutorial: remote backup of local Cloudron backup snapshots with restic / rclone:
Becoming fan of https://ntfy.sh :
Nice find, that's totally self-hostable, you should add it to the list!
We had several instances where we needed a way to have notifications for cloudron apps/events. This would do the trick!
It can also be made a feature request for box notifications to support an API or curl call for admins.
-
As suggested here, I turned my old post into a separate topic:
This is what I use for remote backups of my local Cloudron backup snapshots (done by rsync) via restic / rclone to Onedrive.
restic is a robust backup solution for incremental, encrypted, mountable(!) backups to local and remote storage. rclone, an equally robust sync software, is just a "transporter tool" that expands the available remote storages by a lot.
Maybe it can be a starting point and some inspiration for your personal needs.
Tools
- rclone: https://rclone.org/docs/
- restic: https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#other-services-via-rclone
- ssmtp: https://wiki.archlinux.org/title/SSMTP
Installation
- Install tools above via apt
- afterwards update to latest version (repo versions are old):
sudo restic self-update && sudo rclone selfupdate
Setup rclone
- Enter an interactive setup process via
rclone config - in my case I use Onedrive as it has 1TB of space coming with my Office 365 subscription
- for the rest of this summary, we assume you gave it the repository name "REPOSITORY"
- details at https://rclone.org/commands/rclone_config/
Setup restic
- set up a backup repository
restic -r rclone:REPOSITORY init(for compression support add--repository-version 2- recommended!) - for a subfolder on onedrive just use
restic -r rclone:REPOSITORY:subfolder init(for compression support add--repository-version 2- recommended!) - save password that you gave the repository in file
/home/USER/resticpw - details at https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#other-services-via-rclone
Setup SSMTP
- for receiving backup results, otherwise not needed
- See https://wiki.archlinux.org/title/SSMTP
Cloudron Backup settings
- Provider: mountpoint
- Location:
/media/CloudronBackup(<-- obviously adjust to your settings) - this creates a snapshot at
/media/CloudronBackup/snapshotfor the current backup - Storage Format: rsync
- Adjust schedule and retention to your liking
Backup, Prune and Check scripts
restic-cron-backup.sh: The actual backup#!/bin/bash d=$(date +%Y-%m-%d) if pidof -o %PPID -x “$0”; then echo “$(date “+%d.%m.%Y %T”) Exit, already running.” exit 1 fi restic -r rclone:REPOSITORY:subfolder backup /media/CloudronBackup/snapshot -p=/home/USER/resticpw restic -r rclone:REPOSITORY:subfolder forget --keep-monthly 12 --keep-weekly 5 --keep-daily 14 -p=/home/USER/resticpw restic -r rclone:REPOSITORY:subfolder check --read-data-subset=2% -p=/home/USER/resticpw exitFirst line does the backup (incremental, encrypted), second line is the backup retention, third line checks a random 2 % of all data for errors.
Note that I only backup the/snapshotfolder as all versioning is done by restic.
For compression, add--compression auto(ormax) to thebackupcommand.restic-cron-prune.sh: Pruning unused files in the backup#!/bin/bash d=$(date +%Y-%m-%d) if pidof -o %PPID -x “$0”; then echo “$(date “+%d.%m.%Y %T”) Exit, already running.” exit 1 fi restic -r rclone:REPOSITORY:subfolder prune -p=/home/USER/resticpw exitremoves unused data from the repository, I run this once a week
restic-cron-check.sh: thorough health check of the backups#!/bin/bash d=$(date +%Y-%m-%d) if pidof -o %PPID -x “$0”; then echo “$(date “+%d.%m.%Y %T”) Exit, already running.” exit 1 fi restic -r rclone:REPOSITORY:subfolder check --read-data -p=/home/USER/resticpw exitchecks all data for errors, I run this once a week
Crontab
30 2 * * * sh /home/USER/restic-cron-backup.sh | mailx -s "Restic Backup Results" server@mydomain.com 1 5 1 * * sh /home/USER/restic-cron-prune.sh | mailx -s "Restic Prune Results" server@mydomain.com 1 8 1 * * sh /home/USER/restic-cron-check.sh | mailx -s "Restic Full Check Results" server@mydomain.comBackup daily at 2:30, prune and check once a week. Receive results to specified mail
Mount backups
Just to be complete: You can mount restic backups locally like
restic -r rclone:REPOSITORY:subfolder mount /media/resticmount/ -p=/home/USER/resticpw && cd /media/resticmount
obviously adjust/media/resticmount/to your settings; allows you to browse and copy from full snapshots for each backupList backups
For listing all available snapshots use
restic -r rclone:REPOSITORY:subfolder snapshots -p=/home/USER/resticpwMigrate existing backups to compressed backups
For migrating existing repos to compressed repos use these two steps (will take long!)
restic -r rclone:REPOSITORY:subfolder migrate upgrade_repo_v2 -p=/home/USER/resticpwrestic -r rclone:REPOSITORY:subfolder prune --repack-uncompressed -p=/home/USER/resticpw
See https://restic.readthedocs.io/en/latest/045_working_with_repos.html#upgrading-the-repository-format-version for details.
added command to list all available snapshots to original post
-
added command to list all available snapshots to original post
Note that (very effective) compression using zstd has been added to restic beta recently (beta versions are quite stable) - my repository went from 221 GB down to 180 GB with default settings. I have updated the tutorial to reflect this and also a section for migration of existing repos.
-
Very handy, bookmarking this post for future learning.
Also handy for Rsync with OSX: https://github.com/rsyncOSX/RsyncOSX
-
As suggested here, I turned my old post into a separate topic:
This is what I use for remote backups of my local Cloudron backup snapshots (done by rsync) via restic / rclone to Onedrive.
restic is a robust backup solution for incremental, encrypted, mountable(!) backups to local and remote storage. rclone, an equally robust sync software, is just a "transporter tool" that expands the available remote storages by a lot.
Maybe it can be a starting point and some inspiration for your personal needs.
Tools
- rclone: https://rclone.org/docs/
- restic: https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#other-services-via-rclone
- ssmtp: https://wiki.archlinux.org/title/SSMTP
Installation
- Install tools above via apt
- afterwards update to latest version (repo versions are old):
sudo restic self-update && sudo rclone selfupdate
Setup rclone
- Enter an interactive setup process via
rclone config - in my case I use Onedrive as it has 1TB of space coming with my Office 365 subscription
- for the rest of this summary, we assume you gave it the repository name "REPOSITORY"
- details at https://rclone.org/commands/rclone_config/
Setup restic
- set up a backup repository
restic -r rclone:REPOSITORY init(for compression support add--repository-version 2- recommended!) - for a subfolder on onedrive just use
restic -r rclone:REPOSITORY:subfolder init(for compression support add--repository-version 2- recommended!) - save password that you gave the repository in file
/home/USER/resticpw - details at https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#other-services-via-rclone
Setup SSMTP
- for receiving backup results, otherwise not needed
- See https://wiki.archlinux.org/title/SSMTP
Cloudron Backup settings
- Provider: mountpoint
- Location:
/media/CloudronBackup(<-- obviously adjust to your settings) - this creates a snapshot at
/media/CloudronBackup/snapshotfor the current backup - Storage Format: rsync
- Adjust schedule and retention to your liking
Backup, Prune and Check scripts
restic-cron-backup.sh: The actual backup#!/bin/bash d=$(date +%Y-%m-%d) if pidof -o %PPID -x “$0”; then echo “$(date “+%d.%m.%Y %T”) Exit, already running.” exit 1 fi restic -r rclone:REPOSITORY:subfolder backup /media/CloudronBackup/snapshot -p=/home/USER/resticpw restic -r rclone:REPOSITORY:subfolder forget --keep-monthly 12 --keep-weekly 5 --keep-daily 14 -p=/home/USER/resticpw restic -r rclone:REPOSITORY:subfolder check --read-data-subset=2% -p=/home/USER/resticpw exitFirst line does the backup (incremental, encrypted), second line is the backup retention, third line checks a random 2 % of all data for errors.
Note that I only backup the/snapshotfolder as all versioning is done by restic.
For compression, add--compression auto(ormax) to thebackupcommand.restic-cron-prune.sh: Pruning unused files in the backup#!/bin/bash d=$(date +%Y-%m-%d) if pidof -o %PPID -x “$0”; then echo “$(date “+%d.%m.%Y %T”) Exit, already running.” exit 1 fi restic -r rclone:REPOSITORY:subfolder prune -p=/home/USER/resticpw exitremoves unused data from the repository, I run this once a week
restic-cron-check.sh: thorough health check of the backups#!/bin/bash d=$(date +%Y-%m-%d) if pidof -o %PPID -x “$0”; then echo “$(date “+%d.%m.%Y %T”) Exit, already running.” exit 1 fi restic -r rclone:REPOSITORY:subfolder check --read-data -p=/home/USER/resticpw exitchecks all data for errors, I run this once a week
Crontab
30 2 * * * sh /home/USER/restic-cron-backup.sh | mailx -s "Restic Backup Results" server@mydomain.com 1 5 1 * * sh /home/USER/restic-cron-prune.sh | mailx -s "Restic Prune Results" server@mydomain.com 1 8 1 * * sh /home/USER/restic-cron-check.sh | mailx -s "Restic Full Check Results" server@mydomain.comBackup daily at 2:30, prune and check once a week. Receive results to specified mail
Mount backups
Just to be complete: You can mount restic backups locally like
restic -r rclone:REPOSITORY:subfolder mount /media/resticmount/ -p=/home/USER/resticpw && cd /media/resticmount
obviously adjust/media/resticmount/to your settings; allows you to browse and copy from full snapshots for each backupList backups
For listing all available snapshots use
restic -r rclone:REPOSITORY:subfolder snapshots -p=/home/USER/resticpwMigrate existing backups to compressed backups
For migrating existing repos to compressed repos use these two steps (will take long!)
restic -r rclone:REPOSITORY:subfolder migrate upgrade_repo_v2 -p=/home/USER/resticpwrestic -r rclone:REPOSITORY:subfolder prune --repack-uncompressed -p=/home/USER/resticpw
See https://restic.readthedocs.io/en/latest/045_working_with_repos.html#upgrading-the-repository-format-version for details.
-
@necrevistonnezr Very in-depth. I don't use Cloudron anymore, but this is gonna be really good for admins looking for something easy to manage.
@murgero said in Tutorial: remote backup of local Cloudron backup snapshots with restic / rclone:
@necrevistonnezr Very in-depth. I don't use Cloudron anymore, but this is gonna be really good for admins looking for something easy to manage.
Thanks! Can I ask why you’re not using Cloudron anymore…?
-
@murgero said in Tutorial: remote backup of local Cloudron backup snapshots with restic / rclone:
@necrevistonnezr Very in-depth. I don't use Cloudron anymore, but this is gonna be really good for admins looking for something easy to manage.
Thanks! Can I ask why you’re not using Cloudron anymore…?
@necrevistonnezr No bad reasoning here, just needed a few more features and found something else to use. Cloudron is and always will have a special spot in my heart. Love the Cloudron team, and I'll still be around the forums too
--
https://urgero.org
~ Professional Nerd. Freelance Programmer. ~ -
@necrevistonnezr No bad reasoning here, just needed a few more features and found something else to use. Cloudron is and always will have a special spot in my heart. Love the Cloudron team, and I'll still be around the forums too
@murgero said in Tutorial: remote backup of local Cloudron backup snapshots with restic / rclone:
just needed a few more features and found something else to use.
What are you using now?
-
@murgero said in Tutorial: remote backup of local Cloudron backup snapshots with restic / rclone:
just needed a few more features and found something else to use.
What are you using now?
-
@jdaviescoates Technically speaking I still use Cloudron, just not for my email. I should have made that clear.
Currently using MailCow.
@murgero said in Tutorial: remote backup of local Cloudron backup snapshots with restic / rclone:
Currently using MailCow.
Just out of interest, which features does MailCow have that Cloudron's email offering lacks? Thanks!
-
@murgero said in Tutorial: remote backup of local Cloudron backup snapshots with restic / rclone:
Currently using MailCow.
Just out of interest, which features does MailCow have that Cloudron's email offering lacks? Thanks!
-
@jdaviescoates domain admins, quotas, oauth for 3rd party apps, and rspamd to name a few
@murgero Worth logging each of them as feature requests here given your good experience with both?
-
This is super helpful. I’m hoping we see something like this as a feature baked in though. Like multiple backup sources with different schedules
-
N necrevistonnezr referenced this topic on
-
M michaelpope referenced this topic on
-
-
-
