Cloudron+ZFS?

[LoudLemur]

@infogulch Ubuntu offers ZFS as an experimental option during installation on the Focal Fossa 20.04 LTS release. If you e.g. setup a Virtual Machine with ZFS, and try and install Cloudron, you receive the following error message:

Error: Cloudron requires '/' to be ext4

I suppose the same thing will happen if you chose btrfs for the file system.

[robi]

@LoudLemur removing this restriction would be useful.

[girish]

@LoudLemur We use the overlay2 device driver in Docker. In the past, this device driver did not support btrfs atleast. See https://git.cloudron.io/cloudron/box/-/issues/364 . You can remove the check in the cloudron-setup script and see what happens.

[infogulch]

There is a zfs storage driver that I think is relevant here: https://docs.docker.com/storage/storagedriver/zfs-driver/

[3246]

Thanks for this interesting discussion. I am struggling with backup due to the volume (350/400GB) and wonder if having ZFS+snapshots would be better (faster, easier and more reliable) than Cloudron's way (tar or rsync)?

The new box is a Hetzner dedi with 2x 3TB and a 512GB NVMe. Instead of sRAID1 I am now pondering ZFS mirror with daily snapshots send to the Storagebox and perhaps part of the NVMe as a ZIL log.

I am keen to hear any thoughts and experience you may have folks

[robi]

@3246 that would auto dedupe at a file level, but the main issue is that backups would make it worse due to the tar and compression. Uncompressed you'd save way more space as most files would not change much.

That's why newer tech storage appliances tend to dedupe at a multiple of 4k bytes to be able to dedupe even more regardless of file type or compression type.

Otherwise one could play with storing rapidly or increasingly changing data differently than more static app data, which is exactly that Cloudron does, separating the two.

[necrevistonnezr]

@3246 I have a similar amount of data - pointing Cloudron‘s backup to a local drive via rsync and then pushing this snapshot via restic (and rclone) to Onedrive (via cron) works great. It’s encrypted, de-duped and mountable. Backup results are sent via email. If that’s interesting, i could share my setup.

[robi]

@necrevistonnezr Please do.

[necrevistonnezr]

@robi
I use

Tools

• rclone: https://rclone.org/docs/
• restic: https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#other-services-via-rclone
• ssmtp: https://wiki.archlinux.org/title/SSMTP

Installation

• Install tools above via apt
• afterwards update to latest version (repo versions are old): sudo restic self-update && sudo rclone selfupdate

Setup rclone

• Enter an interactive setup process via rclone config
• in my case I use Onedrive as it has 1TB of space coming with my Office 365 subscription
• for the rest of this summary, we assume you gave it the repository name "REPOSITORY"
• details at https://rclone.org/commands/rclone_config/

Setup restic

• set up a backup repository restic -r rclone:REPOSITORY init
• for a subfolder on onedrive just use restic -r rclone:REPOSITORY:subfolder init
• save password that you gave the repository in file /home/USER/resticpw
• details at https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#other-services-via-rclone

Setup SSMTP

• for receiving backup results, otherwise not needed
• See https://wiki.archlinux.org/title/SSMTP

Cloudron Backup settings

• Provider: mountpoint
• Location: /media/CloudronBackup (<-- obviously adjust to your settings)
• this creates a snapshot at /media/CloudronBackup/snapshot for the current backup
• Storage Format: rsync
• Adjust schedule and retention to your liking

Backup, Prune and Check scripts

restic-cron-backup.sh

#!/bin/bash
d=$(date +%Y-%m-%d)
if pidof -o %PPID -x “$0”; then
echo “$(date “+%d.%m.%Y %T”) Exit, already running.”
exit 1
fi
restic -r rclone:REPOSITORY:subfolder backup /media/CloudronBackup/snapshot -p=/home/USER/resticpw
restic -r rclone:REPOSITORY:subfolder forget --keep-monthly 12 --keep-weekly 5 --keep-daily 14 -p=/home/USER/resticpw
restic -r rclone:REPOSITORY:subfolder check --read-data-subset=2% -p=/home/USER/resticpw
exit

First line does the backup (incremental, encrypted), second line is the backup retention, third line checks a random 2 % of all data for errors

restic-cron-prune.sh

#!/bin/bash
d=$(date +%Y-%m-%d)
if pidof -o %PPID -x “$0”; then
echo “$(date “+%d.%m.%Y %T”) Exit, already running.”
exit 1
fi
restic -r rclone:REPOSITORY:subfolder prune -p=/home/USER/resticpw
exit

removes unused data from the repository, I run this once a week

restic-cron-check.sh

#!/bin/bash
d=$(date +%Y-%m-%d)
if pidof -o %PPID -x “$0”; then
echo “$(date “+%d.%m.%Y %T”) Exit, already running.”
exit 1
fi
restic -r rclone:REPOSITORY:subfolder check --read-data -p=/home/USER/resticpw
exit

checks all data for errors, I run this once a week

Crontab

30 2 * * * sh /home/USER/restic-cron-backup.sh | mailx -s "Restic Backup Results" server@mydomain.com
1 5 1 * * sh /home/USER/restic-cron-prune.sh | mailx -s "Restic Prune Results" server@mydomain.com
1 8 1 * * sh /home/USER/restic-cron-check.sh | mailx -s "Restic Full Check Results" server@mydomain.com

Backup daily at 2:30, prune and check once a week. Receive results to specified mail

Mount backups

Just to be complete: You can mount restic backups locally like
restic -r rclone:REPOSITORY:subfolder mount /media/resticmount/ -p=/home/USER/resticpw && cd /media/resticmount
obviously adjust /media/resticmount/to your settings; allows you to browse and copy from full snapshots for each backup

[robi]

@necrevistonnezr beautiful

Funny how you have to prune cuz it forgets.

[necrevistonnezr]

@robi said in Cloudron+ZFS?:

Funny how you have to prune cuz it forgets.

[3246]

@necrevistonnezr amazing! Thank you for sharing

I am pondering a similar approach and currently back up to a secondary internal drive via rsync without encryption, although the backup drive is using LUKS.

I wonder if using rclone crypt instead of encrypting the files via restic would gain any advantages (e.g. maximum file / folder names, depth, speed)?

Cloudron > rsync to local drive > rclone/crypt via restic > remote

[3246]

Just reading up on restic and encryption, etc and may just skip the rclone part as I am looking to either go to Wasabi or Hetzner Storagebox.

However, I kinda like the crypt part and am looking for any comparisons between rclone/crypt with restic and restic w/ encryption in terms of time it takes to backup and any drawbacks.

[necrevistonnezr]

@3246 said in Cloudron+ZFS?:

Just reading up on restic and encryption, etc and may just skip the rclone part as I am looking to either go to Wasabi or Hetzner Storagebox.

However, I kinda like the crypt part and am looking for any comparisons between rclone/crypt with restic and restic w/ encryption in terms of time it takes to backup and any drawbacks.

Just to be clear: The encryption (as well as deduplication, the repository, data integrity checks, etc.) is completely handled by restic. rclone is just the "transporter tool" that copies data to providers that the restic does not handle (restic out-of-the-box currently handles SFTP, REST-Server, Minio, Wasabi, etc. see https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html).

[3246]

@necrevistonnezr thank you. Is the encryption always part of restic or optional? It looks like it's baked-in the way repos are build, right?

[necrevistonnezr]

@3246 Yes. There is discussions about it (see https://github.com/restic/restic/issues/1018) but that's actually on of the features of restic: It's encryption done right.
And being able to mount the repositories and directly access them is just fantastic.

[3246]

@necrevistonnezr very interesting. Thank you for sharing that link too. I'll give it a try without the rclone step soon

[3246]

I cribbed off the scripts kindly provided by @necrevistonnezr and am using restic straight to a Hetzner Storagebox.

The first upload is running:

[23:53] 0.59%  427 files 4.571 GiB, total 1344856 files 772.976 GiB, 0 errors ETA 66:59:14

Not quite sure where the 772GB are coming from as the directory is much smaller?

390G    /mnt/local_backups/rsync/2022-05-03-130134-941
1.4G    /mnt/local_backups/rsync/snapshot

[jdaviescoates]

All these posts about backing up locally and then to somewhere else are really useful, but a bit hidden away in this thread about something pretty tenuously related.

Methinks @staff should move some of them into a new thread of their own

[necrevistonnezr]

@3246 Are you backing up /snapshot/ or the parent directory?
/snapshot/ is sufficient for a daily backup as it holds the current status of all files - versioning etc. is done by restic.
Also, how did you calculate these dirsizes?