Restoring an app from an rsync backup downloads everything again, even when almost all files are already there

[imc67]

This morning the Immich update to 1.102.3 broke machine learning (topic 16034), so I restored the backup from just before the update. That brought back something I've raised before: backups are fast, restores are very slow.

Immich, sshfs + rsync to a Hetzner Storage Box	Files	Data	Time
Backup this morning (incremental)	151,807	1.1 GB transferred	55 s
Restore of the backup before the update	151,807	291 GB	~1.5 h

Almost all of those 291 GB were already on the server and unchanged. The restore first empties the data directory and then fetches every file again, one by one. During that time the app is down, and my phone can't upload photos.

I understand the earlier answers: Cloudron's rsync format isn't real rsync, and most setups use object storage, where comparing with local files isn't straightforward (13362, 13499). But in 13499 girish mentioned that backup integrity with checksums could make it possible to check whether local and remote files are "the same". That is now in place: every backup has a .backupinfo with a sha256 per file.

What I have in mind is simple: don't empty the data directory, but sync the backup into it. Files whose size and sha256 match the .backupinfo stay where they are, only missing or changed files are downloaded, and files that aren't in the backup are removed. In this case that would have been the photos uploaded after the backup, a few changed thumbnails and the database dump, roughly the size of the incremental backup instead of 291 GB. For sshfs targets that support real rsync (a Hetzner Storage Box does), even plain rsync -a --delete from the snapshot would work. The obvious downside is that the restore then has to read and hash the local files first, but on a local disk that is far faster than fetching 150k files one by one.

It matters even more for a full server restore. At this speed my largest server would take well over a day, while the incremental backups take minutes. I've also followed the sshfs read-speed discussion (13852).

Is something like this possible now that the integrity data is there, at least for the rsync format?

[girish]

Thanks for the reminder! Forgot about this. Yeah, so an idea would be to hash the file locally and only download if something changed. There is a CPU vs network optimization. So, it might make sense for large files and avoid re-downloading then again.

[imc67]

Thanks! One data point from today's restore: the time was driven by the number of files, not their size. The ~100k small thumbnails took most of it, since every file costs a round trip over sshfs, so skipping unchanged small files would help just as much. A quick size + mtime check before hashing would probably keep the CPU side cheap too.