Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse

Cloudron Forum

Apps | Demo | Docs | Install

Tutorial: remote backup of local Cloudron backup snapshots with restic / rclone

Scheduled Pinned Locked Moved Discuss
resticrclonebackups
16 Posts 7 Posters 868 Views
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • necrevistonnezrN Offline
    necrevistonnezrN Offline
    necrevistonnezr
    wrote on last edited by necrevistonnezr
    #1

    As suggested here, I turned my old post into a separate topic:

    This is what I use for remote backups of my local Cloudron backup snapshots (done by rsync) via restic / rclone to Onedrive.

    restic is a robust backup solution for incremental, encrypted, mountable(!) backups to local and remote storage. rclone, an equally robust sync software, is just a "transporter tool" that expands the available remote storages by a lot.

    Maybe it can be a starting point and some inspiration for your personal needs.

    Tools

    • rclone: https://rclone.org/docs/
    • restic: https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#other-services-via-rclone (latest snapshot: https://beta.restic.net/latest/ or directly e.g. https://beta.restic.net/latest_restic_linux_amd64)
    • ssmtp: https://wiki.archlinux.org/title/SSMTP

    Installation

    • Install tools above via apt
    • afterwards update to latest version (repo versions are old): sudo restic self-update && sudo rclone selfupdate (for latest beta use https://beta.restic.net/latest/ or e.g. https://beta.restic.net/latest_restic_linux_amd64)

    Setup rclone

    • Enter an interactive setup process via rclone config
    • in my case I use Onedrive as it has 1TB of space coming with my Office 365 subscription
    • for the rest of this summary, we assume you gave it the repository name "REPOSITORY"
    • details at https://rclone.org/commands/rclone_config/

    Setup restic

    • set up a backup repository restic -r rclone:REPOSITORY init (for compression support add --repository-version 2 if you use https://beta.restic.net/latest/)
    • for a subfolder on onedrive just use restic -r rclone:REPOSITORY:subfolder init (for compression support add --repository-version 2 https://beta.restic.net/latest/)
    • save password that you gave the repository in file /home/USER/resticpw
    • details at https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#other-services-via-rclone

    Setup SSMTP

    • for receiving backup results, otherwise not needed
    • See https://wiki.archlinux.org/title/SSMTP

    Cloudron Backup settings

    • Provider: mountpoint
    • Location: /media/CloudronBackup (<-- obviously adjust to your settings)
    • this creates a snapshot at /media/CloudronBackup/snapshot for the current backup
    • Storage Format: rsync
    • Adjust schedule and retention to your liking

    Backup, Prune and Check scripts

    restic-cron-backup.sh: The actual backup

    #!/bin/bash
    d=$(date +%Y-%m-%d)
    if pidof -o %PPID -x “$0”; then
    echo “$(date “+%d.%m.%Y %T”) Exit, already running.”
    exit 1
    fi
    restic -r rclone:REPOSITORY:subfolder backup /media/CloudronBackup/snapshot -p=/home/USER/resticpw 
    restic -r rclone:REPOSITORY:subfolder forget --keep-monthly 12 --keep-weekly 5 --keep-daily 14 -p=/home/USER/resticpw
    restic -r rclone:REPOSITORY:subfolder check --read-data-subset=2% -p=/home/USER/resticpw
    exit
    

    First line does the backup (incremental, encrypted), second line is the backup retention, third line checks a random 2 % of all data for errors.
    Note that I only backup the /snapshot folder as all versioning is done by restic.
    For compression, add --compression auto (or max) to the backup command.

    restic-cron-prune.sh: Pruning unused files in the backup

    #!/bin/bash
    d=$(date +%Y-%m-%d)
    if pidof -o %PPID -x “$0”; then
    echo “$(date “+%d.%m.%Y %T”) Exit, already running.”
    exit 1
    fi
    restic -r rclone:REPOSITORY:subfolder prune -p=/home/USER/resticpw
    exit
    

    removes unused data from the repository, I run this once a week

    restic-cron-check.sh: thorough health check of the backups

    #!/bin/bash
    d=$(date +%Y-%m-%d)
    if pidof -o %PPID -x “$0”; then
    echo “$(date “+%d.%m.%Y %T”) Exit, already running.”
    exit 1
    fi
    restic -r rclone:REPOSITORY:subfolder check --read-data -p=/home/USER/resticpw
    exit
    

    checks all data for errors, I run this once a week

    Crontab

    30 2 * * * sh /home/USER/restic-cron-backup.sh | mailx -s "Restic Backup Results" server@mydomain.com
    1 5 1 * * sh /home/USER/restic-cron-prune.sh | mailx -s "Restic Prune Results" server@mydomain.com
    1 8 1 * * sh /home/USER/restic-cron-check.sh | mailx -s "Restic Full Check Results" server@mydomain.com
    

    Backup daily at 2:30, prune and check once a week. Receive results to specified mail

    Mount backups

    Just to be complete: You can mount restic backups locally like
    restic -r rclone:REPOSITORY:subfolder mount /media/resticmount/ -p=/home/USER/resticpw && cd /media/resticmount
    obviously adjust /media/resticmount/to your settings; allows you to browse and copy from full snapshots for each backup

    List backups

    For listing all available snapshots use
    restic -r rclone:REPOSITORY:subfolder snapshots -p=/home/USER/resticpw

    Migrate existing backups to compressed backups

    For migrating existing repos to compressed repos use these two steps (will take long!)

    • restic -r rclone:REPOSITORY:subfolder migrate upgrade_repo_v2 -p=/home/USER/resticpw
    • restic -r rclone:REPOSITORY:subfolder prune --repack-uncompressed -p=/home/USER/resticpw

    See https://restic.readthedocs.io/en/latest/045_working_with_repos.html#upgrading-the-repository-format-version for details.

    timconsidineT necrevistonnezrN murgeroM 3 Replies Last reply
    10
  • timconsidineT Offline
    timconsidineT Offline
    timconsidine App Dev
    replied to necrevistonnezr on last edited by
    #2

    @necrevistonnezr nice article.
    Great fan of rclone.
    I use it with wasabi and scaleway.

    Don't know restic so I'm off to learn.

    Becoming fan of https://ntfy.sh : great for adding into scripts run by cron to confirm running/script results/server status (e.g. df -h)

    robiR 1 Reply Last reply
    2
  • robiR Offline
    robiR Offline
    robi
    replied to timconsidine on last edited by
    #3

    @timconsidine said in Tutorial: remote backup of local Cloudron backup snapshots with restic / rclone:

    Becoming fan of https://ntfy.sh :

    Nice find, that's totally self-hostable, you should add it to the list!

    We had several instances where we needed a way to have notifications for cloudron apps/events. This would do the trick!

    It can also be made a feature request for box notifications to support an API or curl call for admins.

    Life of sky tech

    timconsidineT 1 Reply Last reply
    3
  • timconsidineT Offline
    timconsidineT Offline
    timconsidine App Dev
    replied to robi on last edited by
    #4

    @robi I've been meaning to have a go at packaging it.
    But I stumbled on something (I forget what).
    Will try again.
    And will open a wishlist for someone more talented than me to have a crack.

    1 Reply Last reply
    3
  • necrevistonnezrN Offline
    necrevistonnezrN Offline
    necrevistonnezr
    replied to necrevistonnezr on last edited by
    #5

    added command to list all available snapshots to original post

    necrevistonnezrN 1 Reply Last reply
    3
  • necrevistonnezrN Offline
    necrevistonnezrN Offline
    necrevistonnezr
    replied to necrevistonnezr on last edited by necrevistonnezr
    #6

    Note that (very effective) compression using zstd has been added to restic beta recently (beta versions are quite stable) - my repository went from 221 GB down to 180 GB with default settings. I have updated the tutorial to reflect this and also a section for migration of existing repos.

    1 Reply Last reply
    7
  • marcusquinnM Offline
    marcusquinnM Offline
    marcusquinn
    wrote on last edited by
    #7

    Very handy, bookmarking this post for future learning.

    Also handy for Rsync with OSX: https://github.com/rsyncOSX/RsyncOSX

    We're not here for a long time - but we are here for a good time :)
    Jersey/UK
    Work & Ecommerce Advice: https://brandlight.org
    Personal & Software Tips: https://marcusquinn.com

    1 Reply Last reply
    0
  • murgeroM Offline
    murgeroM Offline
    murgero App Dev
    replied to necrevistonnezr on last edited by
    #8

    @necrevistonnezr Very in-depth. I don't use Cloudron anymore, but this is gonna be really good for admins looking for something easy to manage.

    --
    https://urgero.org
    ~ Professional Nerd. Freelance Programmer. ~
    Matrix: @murgero:urgero.org

    necrevistonnezrN 1 Reply Last reply
    0
  • necrevistonnezrN Offline
    necrevistonnezrN Offline
    necrevistonnezr
    replied to murgero on last edited by
    #9

    @murgero said in Tutorial: remote backup of local Cloudron backup snapshots with restic / rclone:

    @necrevistonnezr Very in-depth. I don't use Cloudron anymore, but this is gonna be really good for admins looking for something easy to manage.

    Thanks! Can I ask why you’re not using Cloudron anymore…?

    murgeroM 1 Reply Last reply
    1
  • murgeroM Offline
    murgeroM Offline
    murgero App Dev
    replied to necrevistonnezr on last edited by
    #10

    @necrevistonnezr No bad reasoning here, just needed a few more features and found something else to use. Cloudron is and always will have a special spot in my heart. Love the Cloudron team, and I'll still be around the forums too

    --
    https://urgero.org
    ~ Professional Nerd. Freelance Programmer. ~
    Matrix: @murgero:urgero.org

    jdaviescoatesJ 1 Reply Last reply
    0
  • jdaviescoatesJ Offline
    jdaviescoatesJ Offline
    jdaviescoates
    replied to murgero on last edited by
    #11

    @murgero said in Tutorial: remote backup of local Cloudron backup snapshots with restic / rclone:

    just needed a few more features and found something else to use.

    What are you using now?

    I use Cloudron with Gandi & Hetzner

    murgeroM 1 Reply Last reply
    0
  • murgeroM Offline
    murgeroM Offline
    murgero App Dev
    replied to jdaviescoates on last edited by
    #12

    @jdaviescoates Technically speaking I still use Cloudron, just not for my email. I should have made that clear.

    Currently using MailCow.

    --
    https://urgero.org
    ~ Professional Nerd. Freelance Programmer. ~
    Matrix: @murgero:urgero.org

    jdaviescoatesJ 1 Reply Last reply
    1
  • jdaviescoatesJ Offline
    jdaviescoatesJ Offline
    jdaviescoates
    replied to murgero on last edited by
    #13

    @murgero said in Tutorial: remote backup of local Cloudron backup snapshots with restic / rclone:

    Currently using MailCow.

    Just out of interest, which features does MailCow have that Cloudron's email offering lacks? Thanks!

    I use Cloudron with Gandi & Hetzner

    murgeroM 1 Reply Last reply
    0
  • murgeroM Offline
    murgeroM Offline
    murgero App Dev
    replied to jdaviescoates on last edited by
    #14

    @jdaviescoates domain admins, quotas, oauth for 3rd party apps, and rspamd to name a few

    --
    https://urgero.org
    ~ Professional Nerd. Freelance Programmer. ~
    Matrix: @murgero:urgero.org

    marcusquinnM 1 Reply Last reply
    3
  • marcusquinnM Offline
    marcusquinnM Offline
    marcusquinn
    replied to murgero on last edited by
    #15

    @murgero Worth logging each of them as feature requests here given your good experience with both?

    We're not here for a long time - but we are here for a good time :)
    Jersey/UK
    Work & Ecommerce Advice: https://brandlight.org
    Personal & Software Tips: https://marcusquinn.com

    1 Reply Last reply
    2
  • ? Offline
    ? Offline
    A Former User
    wrote on last edited by
    #16

    This is super helpful. I’m hoping we see something like this as a feature baked in though. Like multiple backup sources with different schedules

    1 Reply Last reply
    3
  • necrevistonnezrN necrevistonnezr referenced this topic on
  • M michaelpope referenced this topic on
  • necrevistonnezrN necrevistonnezr referenced this topic on

  • Login

  • Don't have an account? Register

  • Login or register to search.
  • First post
    Last post
0
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Login

  • Don't have an account? Register

  • Login or register to search.