Long backups, local and remote, failing consistently

[jadudm]

(I have a suspicion that this is a variation on this post from a while back.)

I have configured backups as follows:

backup set	encr?	target	day	time	files	size
bitwarden	Y	storage box	daily	20:00	800	7MB
photos	N	storage box	S	03:00	300K	200GB
photos	N	NAS	Su	03:00	300K	200GB
full (- music, -photos)	Y	NAS	MWF	03:00	18K	12GB
music	N	NAS	T	03:00	?	600GB

What I'm finding is that my Immich (photos) instance does not want to backup. To be more precise: Immich consistently fails a long way into the backup. In both the case where it is talking to a storage box (overseas, for me) and to my local NAS, it is configured as an SSHFS mount. In each location I have set up a folder called $HOME/backups, and used a subpath for each backup (e.g. photos, so that the full path becomes $HOME/backups/photos, $HOME/backups/vaults, etc.). In all cases, I'm using rsync with hardlinks.

I removed the photos (which is large/has many files) and the music from the full backup set, because I want to target them separately for backup. And, I want to make sure my full backup completes.

I can backup the bitwarden instance, because it is small. I have not yet seen the photos complete. I end up somewhere around 290K files, and there's an SSH error that drops. I don't know what the root cause is. (And, I'm now waiting for another backup, because Immich kicked off an update... so, I have to wait.)

I'll update this thread if/when it fails again. Possible root causes (that would be difficult for me to work around):

1. Too many files. I would think rsync would have no problems.
2. Files changing. Immich likes to touch things. Is it paused during backup? If not, could that be the problem? (There are tempfiles that get created as part of its processes; could those be in the set, then get processed/deleted before the backup gets to them, and then it breaks the backup? But, pausing during backups is disruptive/not appropriate for a live system, so... that's not actually a solution path. Ignore me.)
3. Not enough RAM. Do I need to give the backup process more RAM?

The NAS is a TrueNAS (therefore Debian) machine sitting next to the Cloudron host. Neither seems to be under any kind of RAM pressure that I can see. Neither is doing anything else of substance while the backups are happening.

Unrelated: I do not know what happens when Immich updates, because I am targeting it with two backup points. Does that mean an app update will trigger a backup to both locations? Will it do so sequentially, or simultaneously?

possible other solutions

I would like the SSHFS backup to "just work." But, I'm aware of the complexity of the systems involved.

Other solutions I could consider:

1. Use object storage. I don't like this one. When using rsync with many files, I discovered that (on B2) I could end up paying a lot for transactions if I had a frequent backup, because rsync likes to touch so many things. This was the point of getting the NAS.
2. Run my own object storage on the NAS. I really don't want to do that. And, it doesn't solve my off-site photos backup.
3. Introduce JuiceFS on the Cloudron host. I could put JuiceFS on the Cloudron host. I dislike this for all of the obvious reasons. But, it would let me set up an SSHFS mount to my remote host, and Cloudron/rsync would think it was a local filesystem. This might only be pushing the problems downwards, though.
4. Backup locally, and rsync the backup. I think I have the disk space for this. This is probably my most robust answer, but it is... annoying. It means I have to set up a secondary layer of rsync processes. On the other hand, I have confidence that if I set up a local volume, the Cloudron backup will "just work."

Ultimately, I'm trying to figure out how to reliably back things up. I think #4 is my best bet.

[jadudm]

I'm 140k into another run. Took all day... will bump thread with results when there are results...

[jadudm]

The Immich (photos) backup ended as follows.

Feb 10 03:11:21 box:backupformat/rsync sync: adding data/upload/upload/d354571e-1804-4798-bd79-e29690172c14/d9/d7/d9d762ae-5a69-461d-9387-84882f110276.jpg.xmp position 227458 try 1
Feb 10 03:11:21 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/upload/d354571e-1804-4798-bd79-e29690172c14/d9/d7/d9d762ae-5a69-461d-9387-84882f110276.jpg.xmp","reason":"new","position":227458}
Feb 10 03:11:21 Exiting with code 70
Feb 10 03:11:21 box:taskworker Terminated
Feb 10 05:03:04 13:M 10 Feb 2026 10:03:04.004 * 10 changes in 300 seconds. Saving...
Feb 10 05:03:04 13:M 10 Feb 2026 10:03:04.004 * Background saving started by pid 298

I do not know for certain if this was the local or remote backup. Local, the snapshot folder dates Feb 9 03:13, and remote it dates Feb 9, 02:35. Those... appear to be the created times, using ls -ac.

According to logs, my music backup ran at Tuesday at 3AM, and it completed in 1m30s or thereabouts. So, that took place 10m before this failure. The music backup would be against the NAS.

Immich still wants to update.

Are there any thoughts as to what I should consider doing to get to a successful backup of my photos?

Absent a way for Cloudron to successfully backup Immich, I feel like the following are my options:

1. JuiceFS would probably let rsync complete and support hardlinks. I would create an SSHFS mount via Juice from a folder on my localhost -> the target system. Then, I would mount that folder as a local volume (filesystem). As far as Cloudron would be concerned, it would be a regular filesystem. Downside? It's a moving piece in-between me and my files, and a point for data loss.
2. I could use object storage, but I'm concerned about operation costs. An rsync -> object store approach with this many files means... probably hundreds of thousands of API calls for every backup. Depending on the provider, that ends up costing.
3. Use tar? I feel that a tarball is really inefficient, since the photos don't change often/at all.
4. Backup locally and rsync the backup. This would eat disk, but I have space to spare on the Cloudron host; it runs on a mirrored 8TB pair. If I keep three backups (monthly), I would end up with nearly a TB of data, but I could rsync that to the NAS and remote. The rotation would happen locally, I'd get off-site and local backups, and the cost would be that each photo takes 4x the space (original + 3x copies on the local filesystem for rsync rotation).

[jadudm]

I could also use the fstab to mount an SSHFS filesystem to the remotes, and let Cloudron backup via filesystem there. This would move the management of the mount out of the hands of Cloudron, and into the hands of the OS.

I don't know if that would help.