Tutorial: remote backup of local Cloudron backup snapshots with restic / rclone
-
@timconsidine said in Tutorial: remote backup of local Cloudron backup snapshots with restic / rclone:
Becoming fan of https://ntfy.sh :
Nice find, that's totally self-hostable, you should add it to the list!
We had several instances where we needed a way to have notifications for cloudron apps/events. This would do the trick!
It can also be made a feature request for box notifications to support an API or curl call for admins.
-
As suggested here, I turned my old post into a separate topic:
This is what I use for remote backups of my local Cloudron backup snapshots (done by rsync) via restic / rclone to Onedrive.
restic is a robust backup solution for incremental, encrypted, mountable(!) backups to local and remote storage. rclone, an equally robust sync software, is just a "transporter tool" that expands the available remote storages by a lot.
Maybe it can be a starting point and some inspiration for your personal needs.
Tools
- rclone: https://rclone.org/docs/
- restic: https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#other-services-via-rclone
- ssmtp: https://wiki.archlinux.org/title/SSMTP
Installation
- Install tools above via apt
- afterwards update to latest version (repo versions are old):
sudo restic self-update && sudo rclone selfupdate
Setup rclone
- Enter an interactive setup process via
rclone config - in my case I use Onedrive as it has 1TB of space coming with my Office 365 subscription
- for the rest of this summary, we assume you gave it the repository name "REPOSITORY"
- details at https://rclone.org/commands/rclone_config/
Setup restic
- set up a backup repository
restic -r rclone:REPOSITORY init(for compression support add--repository-version 2- recommended!) - for a subfolder on onedrive just use
restic -r rclone:REPOSITORY:subfolder init(for compression support add--repository-version 2- recommended!) - save password that you gave the repository in file
/home/USER/resticpw - details at https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#other-services-via-rclone
Setup SSMTP
- for receiving backup results, otherwise not needed
- See https://wiki.archlinux.org/title/SSMTP
Cloudron Backup settings
- Provider: mountpoint
- Location:
/media/CloudronBackup(<-- obviously adjust to your settings) - this creates a snapshot at
/media/CloudronBackup/snapshotfor the current backup - Storage Format: rsync
- Adjust schedule and retention to your liking
Backup, Prune and Check scripts
restic-cron-backup.sh: The actual backup#!/bin/bash d=$(date +%Y-%m-%d) if pidof -o %PPID -x “$0”; then echo “$(date “+%d.%m.%Y %T”) Exit, already running.” exit 1 fi restic -r rclone:REPOSITORY:subfolder backup /media/CloudronBackup/snapshot -p=/home/USER/resticpw restic -r rclone:REPOSITORY:subfolder forget --keep-monthly 12 --keep-weekly 5 --keep-daily 14 -p=/home/USER/resticpw restic -r rclone:REPOSITORY:subfolder check --read-data-subset=2% -p=/home/USER/resticpw exitFirst line does the backup (incremental, encrypted), second line is the backup retention, third line checks a random 2 % of all data for errors.
Note that I only backup the/snapshotfolder as all versioning is done by restic.
For compression, add--compression auto(ormax) to thebackupcommand.restic-cron-prune.sh: Pruning unused files in the backup#!/bin/bash d=$(date +%Y-%m-%d) if pidof -o %PPID -x “$0”; then echo “$(date “+%d.%m.%Y %T”) Exit, already running.” exit 1 fi restic -r rclone:REPOSITORY:subfolder prune -p=/home/USER/resticpw exitremoves unused data from the repository, I run this once a week
restic-cron-check.sh: thorough health check of the backups#!/bin/bash d=$(date +%Y-%m-%d) if pidof -o %PPID -x “$0”; then echo “$(date “+%d.%m.%Y %T”) Exit, already running.” exit 1 fi restic -r rclone:REPOSITORY:subfolder check --read-data -p=/home/USER/resticpw exitchecks all data for errors, I run this once a week
Crontab
30 2 * * * sh /home/USER/restic-cron-backup.sh | mailx -s "Restic Backup Results" server@mydomain.com 1 5 1 * * sh /home/USER/restic-cron-prune.sh | mailx -s "Restic Prune Results" server@mydomain.com 1 8 1 * * sh /home/USER/restic-cron-check.sh | mailx -s "Restic Full Check Results" server@mydomain.comBackup daily at 2:30, prune and check once a week. Receive results to specified mail
Mount backups
Just to be complete: You can mount restic backups locally like
restic -r rclone:REPOSITORY:subfolder mount /media/resticmount/ -p=/home/USER/resticpw && cd /media/resticmount
obviously adjust/media/resticmount/to your settings; allows you to browse and copy from full snapshots for each backupList backups
For listing all available snapshots use
restic -r rclone:REPOSITORY:subfolder snapshots -p=/home/USER/resticpwMigrate existing backups to compressed backups
For migrating existing repos to compressed repos use these two steps (will take long!)
restic -r rclone:REPOSITORY:subfolder migrate upgrade_repo_v2 -p=/home/USER/resticpwrestic -r rclone:REPOSITORY:subfolder prune --repack-uncompressed -p=/home/USER/resticpw
See https://restic.readthedocs.io/en/latest/045_working_with_repos.html#upgrading-the-repository-format-version for details.
added command to list all available snapshots to original post
-
added command to list all available snapshots to original post
Note that (very effective) compression using zstd has been added to restic beta recently (beta versions are quite stable) - my repository went from 221 GB down to 180 GB with default settings. I have updated the tutorial to reflect this and also a section for migration of existing repos.
-
Very handy, bookmarking this post for future learning.
Also handy for Rsync with OSX: https://github.com/rsyncOSX/RsyncOSX
-
As suggested here, I turned my old post into a separate topic:
This is what I use for remote backups of my local Cloudron backup snapshots (done by rsync) via restic / rclone to Onedrive.
restic is a robust backup solution for incremental, encrypted, mountable(!) backups to local and remote storage. rclone, an equally robust sync software, is just a "transporter tool" that expands the available remote storages by a lot.
Maybe it can be a starting point and some inspiration for your personal needs.
Tools
- rclone: https://rclone.org/docs/
- restic: https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#other-services-via-rclone
- ssmtp: https://wiki.archlinux.org/title/SSMTP
Installation
- Install tools above via apt
- afterwards update to latest version (repo versions are old):
sudo restic self-update && sudo rclone selfupdate
Setup rclone
- Enter an interactive setup process via
rclone config - in my case I use Onedrive as it has 1TB of space coming with my Office 365 subscription
- for the rest of this summary, we assume you gave it the repository name "REPOSITORY"
- details at https://rclone.org/commands/rclone_config/
Setup restic
- set up a backup repository
restic -r rclone:REPOSITORY init(for compression support add--repository-version 2- recommended!) - for a subfolder on onedrive just use
restic -r rclone:REPOSITORY:subfolder init(for compression support add--repository-version 2- recommended!) - save password that you gave the repository in file
/home/USER/resticpw - details at https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#other-services-via-rclone
Setup SSMTP
- for receiving backup results, otherwise not needed
- See https://wiki.archlinux.org/title/SSMTP
Cloudron Backup settings
- Provider: mountpoint
- Location:
/media/CloudronBackup(<-- obviously adjust to your settings) - this creates a snapshot at
/media/CloudronBackup/snapshotfor the current backup - Storage Format: rsync
- Adjust schedule and retention to your liking
Backup, Prune and Check scripts
restic-cron-backup.sh: The actual backup#!/bin/bash d=$(date +%Y-%m-%d) if pidof -o %PPID -x “$0”; then echo “$(date “+%d.%m.%Y %T”) Exit, already running.” exit 1 fi restic -r rclone:REPOSITORY:subfolder backup /media/CloudronBackup/snapshot -p=/home/USER/resticpw restic -r rclone:REPOSITORY:subfolder forget --keep-monthly 12 --keep-weekly 5 --keep-daily 14 -p=/home/USER/resticpw restic -r rclone:REPOSITORY:subfolder check --read-data-subset=2% -p=/home/USER/resticpw exitFirst line does the backup (incremental, encrypted), second line is the backup retention, third line checks a random 2 % of all data for errors.
Note that I only backup the/snapshotfolder as all versioning is done by restic.
For compression, add--compression auto(ormax) to thebackupcommand.restic-cron-prune.sh: Pruning unused files in the backup#!/bin/bash d=$(date +%Y-%m-%d) if pidof -o %PPID -x “$0”; then echo “$(date “+%d.%m.%Y %T”) Exit, already running.” exit 1 fi restic -r rclone:REPOSITORY:subfolder prune -p=/home/USER/resticpw exitremoves unused data from the repository, I run this once a week
restic-cron-check.sh: thorough health check of the backups#!/bin/bash d=$(date +%Y-%m-%d) if pidof -o %PPID -x “$0”; then echo “$(date “+%d.%m.%Y %T”) Exit, already running.” exit 1 fi restic -r rclone:REPOSITORY:subfolder check --read-data -p=/home/USER/resticpw exitchecks all data for errors, I run this once a week
Crontab
30 2 * * * sh /home/USER/restic-cron-backup.sh | mailx -s "Restic Backup Results" server@mydomain.com 1 5 1 * * sh /home/USER/restic-cron-prune.sh | mailx -s "Restic Prune Results" server@mydomain.com 1 8 1 * * sh /home/USER/restic-cron-check.sh | mailx -s "Restic Full Check Results" server@mydomain.comBackup daily at 2:30, prune and check once a week. Receive results to specified mail
Mount backups
Just to be complete: You can mount restic backups locally like
restic -r rclone:REPOSITORY:subfolder mount /media/resticmount/ -p=/home/USER/resticpw && cd /media/resticmount
obviously adjust/media/resticmount/to your settings; allows you to browse and copy from full snapshots for each backupList backups
For listing all available snapshots use
restic -r rclone:REPOSITORY:subfolder snapshots -p=/home/USER/resticpwMigrate existing backups to compressed backups
For migrating existing repos to compressed repos use these two steps (will take long!)
restic -r rclone:REPOSITORY:subfolder migrate upgrade_repo_v2 -p=/home/USER/resticpwrestic -r rclone:REPOSITORY:subfolder prune --repack-uncompressed -p=/home/USER/resticpw
See https://restic.readthedocs.io/en/latest/045_working_with_repos.html#upgrading-the-repository-format-version for details.
-
@necrevistonnezr Very in-depth. I don't use Cloudron anymore, but this is gonna be really good for admins looking for something easy to manage.
@murgero said in Tutorial: remote backup of local Cloudron backup snapshots with restic / rclone:
@necrevistonnezr Very in-depth. I don't use Cloudron anymore, but this is gonna be really good for admins looking for something easy to manage.
Thanks! Can I ask why you’re not using Cloudron anymore…?
-
@murgero said in Tutorial: remote backup of local Cloudron backup snapshots with restic / rclone:
@necrevistonnezr Very in-depth. I don't use Cloudron anymore, but this is gonna be really good for admins looking for something easy to manage.
Thanks! Can I ask why you’re not using Cloudron anymore…?
@necrevistonnezr No bad reasoning here, just needed a few more features and found something else to use. Cloudron is and always will have a special spot in my heart. Love the Cloudron team, and I'll still be around the forums too
--
https://urgero.org
~ Professional Nerd. Freelance Programmer. ~ -
@necrevistonnezr No bad reasoning here, just needed a few more features and found something else to use. Cloudron is and always will have a special spot in my heart. Love the Cloudron team, and I'll still be around the forums too
@murgero said in Tutorial: remote backup of local Cloudron backup snapshots with restic / rclone:
just needed a few more features and found something else to use.
What are you using now?
-
@murgero said in Tutorial: remote backup of local Cloudron backup snapshots with restic / rclone:
just needed a few more features and found something else to use.
What are you using now?
-
@jdaviescoates Technically speaking I still use Cloudron, just not for my email. I should have made that clear.
Currently using MailCow.
@murgero said in Tutorial: remote backup of local Cloudron backup snapshots with restic / rclone:
Currently using MailCow.
Just out of interest, which features does MailCow have that Cloudron's email offering lacks? Thanks!
-
@murgero said in Tutorial: remote backup of local Cloudron backup snapshots with restic / rclone:
Currently using MailCow.
Just out of interest, which features does MailCow have that Cloudron's email offering lacks? Thanks!
-
@jdaviescoates domain admins, quotas, oauth for 3rd party apps, and rspamd to name a few
@murgero Worth logging each of them as feature requests here given your good experience with both?
-
This is super helpful. I’m hoping we see something like this as a feature baked in though. Like multiple backup sources with different schedules
-
N necrevistonnezr referenced this topic on
-
M michaelpope referenced this topic on
-
-
-
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login