Long backups, local and remote, failing consistently

[jadudm]

The Immich (photos) backup ended as follows.

Feb 10 03:11:21 box:backupformat/rsync sync: adding data/upload/upload/d354571e-1804-4798-bd79-e29690172c14/d9/d7/d9d762ae-5a69-461d-9387-84882f110276.jpg.xmp position 227458 try 1
Feb 10 03:11:21 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/upload/d354571e-1804-4798-bd79-e29690172c14/d9/d7/d9d762ae-5a69-461d-9387-84882f110276.jpg.xmp","reason":"new","position":227458}
Feb 10 03:11:21 Exiting with code 70
Feb 10 03:11:21 box:taskworker Terminated
Feb 10 05:03:04 13:M 10 Feb 2026 10:03:04.004 * 10 changes in 300 seconds. Saving...
Feb 10 05:03:04 13:M 10 Feb 2026 10:03:04.004 * Background saving started by pid 298

I do not know for certain if this was the local or remote backup. Local, the snapshot folder dates Feb 9 03:13, and remote it dates Feb 9, 02:35. Those... appear to be the created times, using ls -ac.

According to logs, my music backup ran at Tuesday at 3AM, and it completed in 1m30s or thereabouts. So, that took place 10m before this failure. The music backup would be against the NAS.

Immich still wants to update.

Are there any thoughts as to what I should consider doing to get to a successful backup of my photos?

Absent a way for Cloudron to successfully backup Immich, I feel like the following are my options:

1. JuiceFS would probably let rsync complete and support hardlinks. I would create an SSHFS mount via Juice from a folder on my localhost -> the target system. Then, I would mount that folder as a local volume (filesystem). As far as Cloudron would be concerned, it would be a regular filesystem. Downside? It's a moving piece in-between me and my files, and a point for data loss.
2. I could use object storage, but I'm concerned about operation costs. An rsync -> object store approach with this many files means... probably hundreds of thousands of API calls for every backup. Depending on the provider, that ends up costing.
3. Use tar? I feel that a tarball is really inefficient, since the photos don't change often/at all.
4. Backup locally and rsync the backup. This would eat disk, but I have space to spare on the Cloudron host; it runs on a mirrored 8TB pair. If I keep three backups (monthly), I would end up with nearly a TB of data, but I could rsync that to the NAS and remote. The rotation would happen locally, I'd get off-site and local backups, and the cost would be that each photo takes 4x the space (original + 3x copies on the local filesystem for rsync rotation).

[jadudm]

I could also use the fstab to mount an SSHFS filesystem to the remotes, and let Cloudron backup via filesystem there. This would move the management of the mount out of the hands of Cloudron, and into the hands of the OS.

I don't know if that would help.

[jadudm]

@james , do you have any thoughts?

I had to reboot the server for updates yesterday; as a result, the Immich app is (again) trying to backup. It is now 14K into another attempt. I have every belief that it will fail some 250K files into the backup.

Do any of the strategies I've brainstormed sound better than the others from y'alls perspective?

We can leave this thread open as I explore, but I think the answer is "I can't backup my photos by simply adding an SSHFS backup location." I apparently have to solve this some other way.

[james]

Hello @jadudm

@jadudm said in Long backups, local and remote, failing consistently:

Unrelated: I do not know what happens when Immich updates, because I am targeting it with two backup points. Does that mean an app update will trigger a backup to both locations? Will it do so sequentially, or simultaneously?

This depends on the backup site configuration.
You can configure what backup site should back up.

@jadudm said in Long backups, local and remote, failing consistently:

Feb 10 03:11:21 Exiting with code 70

This could indicate a lack of memory so yes maybe increasing the backup memory size could help.

---

Since SSHFS with RSYNC backups are incremental I am also wondering why it does take to long.

A question about your local TrueNAS connection.
Your Cloudron server is hosted with a provider like Hetzner or is it also hosted locally in the same network as your TrueNAS?
If hosted in the Cloud, maybe the connection between the Cloudron server and your local TrueNAS has some issue.
This could be anything from your internet service provider throtteling the connection, DNS issues, maybe even hardware issues like the network interface and cable on the TrueNAS.
I assume your TrueNAS has a default 1GBit port and cable and is connected to the router directly.
If so, maybe checking the network interface on the TrueNAS is really using 1 GBit and not using a fallback to 100 Mbit due to a bad cable.

[jadudm]

Good questions. The configuration locally is that the machines all live behind an OpnSense router. Cloudron is hosted on a VM on a small machine (and has 24GB of RAM allocated to it, and does not show signs of RAM pressure), and the NAS itself is running TrueNAS w/ 40GB of RAM available (it is never under RAM pressure, as far as I can tell).

cloudron.lan -> switch -> nas.lan

Both machines are local. The cables could be poor; I can check. This is why I think the SSHFS failure on the Cloudron -> NAS connection is so worrying; there's no good reason why it should fail, from what I can tell.

I can... understand that the SSHFS backup to the storage box might be troublesome, given the distances involved. The local connection, though, should "just work."

I'll dig more into possible memory issues.

[james]

Hello @jadudm

Also increase your memory limit for the backup site setting.

[jadudm]

Interesting. I think I had missed that setting before.

I tried two things, but now need to head to work.

I created a SMB share on the NAS. I was able to establish a backup site... and, I just re-created an SSHFS mount per above, and gave it 6GB of RAM.

Feb 11 09:16:30 box:taskworker Starting task 9902. Logs are at /home/yellowtent/platformdata/logs/tasks/9902.log
Feb 11 09:16:30 box:taskworker Running task of type backup
Feb 11 09:16:30 box:backuptask fullBackup: skipped backup ...
Feb 11 09:16:30 box:tasks updating task 9902 with: {"percent":66.38461538461539,"message":"Backing up photos.jadud.com (17/23). Waiting for lock"}
Feb 11 09:16:30 box:locks write: current locks: {"full_backup_task_846414c7-0abc-4ae1-8432-2430e5008342":null,"app_backup_a6dc2056-829f-46c4-bf31-7a93cba4af11":"9902"}
Feb 11 09:16:30 box:locks acquire: app_backup_a6dc2056-829f-46c4-bf31-7a93cba4af11
Feb 11 09:16:30 box:backuptask fullBackup: app photos.jadud.com backup finished. Took 0.002 seconds
Feb 11 09:16:30 box:locks write: current locks: {"full_backup_task_846414c7-0abc-4ae1-8432-2430e5008342":null}
Feb 11 09:16:30 box:locks release: app_backup_a6dc2056-829f-46c4-bf31-7a93cba4af11
Feb 11 09:16:30 box:backuptask fullBackup: skipped backup ...
Feb 11 09:16:30 box:tasks setCompleted - 9902: {"result":[],"error":null,"percent":100}
Feb 11 09:16:30 box:tasks updating task 9902 with: {"completed":true,"result":[],"error":null,"percent":100}
Feb 11 09:16:30 box:taskworker Task took 0.066 seconds
Feb 11 09:16:30 Exiting with code 0

If I try and kick off the backup, it starts up and exits immediately. Is there a lock floating somewhere? (Is that the full backup task lock?)

No backups are running that I can see, but this is now a new behavior. I have rebooted the machine, and this does not change.

No doubt, I've created this problem through my iterations.

[jadudm]

OK. Solution so far:

1. I removed all backup sites and rebooted. (There's a question at the end.)
2. I added a CIFS point (instead of SSHFS) to the local NAS.
3. Gave the backup 5GB of RAM, and set the concurrency to 100
4. Waited an hour or two. Two? What is time.

The backup for Immich succeeded.

I may try an SSHFS backup with similar parameters, but I'll... be limited on the storage box with regards to concurrency. So, we'll see.

QUESTION: I have noticed when app backups fail, there's sometimes a stale lock. Where is that lock? I would like to be able to remove the lock without having to reboot. Is it in the DB? A file? Where does Box keep those app backup locks?

I'm not convinced I've solved my problem, but I'm starting to think the RAM for the backup(s) may matter, which I had never encountered before.

[james]

Hello @jadudm

@jadudm said in Long backups, local and remote, failing consistently:

QUESTION: I have noticed when app backups fail, there's sometimes a stale lock. Where is that lock? I would like to be able to remove the lock without having to reboot. Is it in the DB? A file? Where does Box keep those app backup locks?

A failing app backup should result in the lock still being released.
If this occours again, can you please inform me here so we can take a look?

[jadudm]

Will do, James. I have not been able to recreate the held lock issue. I was starting/stopping jobs a fair bit at one point, and can't... be precise about where in the backup cycle those cancellations happened that a cleanup might not have happened. I will watch for it in the future.

When I said there was no RAM pressure, I meant that was true for the server. However, my jobs all had 1GB of RAM. Your suggestion clued me in; because that value must be set after you setup the backup job, I had never noticed it before... or, not realized how critical it might be. I have bumped them all to 6GB of RAM, and so far, I've been seeing backup successes.

Barring the question below, I'd say we could close this issue. The lesson learned is that I need to provide my backup tasks more RAM. Because I have some RAM to spare, I'm going aggressive, and giving things 6GB. I did not attempt to settle on a smaller amount, for anyone who comes along after--- I just gave the tasks a limit that I considered to be "a lot" in this context.

I still see some things like the errors below. The backup completes successfully, but I'm unclear why there would be errors like these sprinkled throughout the backup. Is the relative path full/snapshot/app_... actually correct? Or, should that be a full path (e.g. the base path I provided at setup time along with the relative path)? In the command that succeeds, it is a full path.

Feb 13 16:11:00 box:shell filesystem: ssh -o "StrictHostKeyChecking no" -i /tmp/identity_file_d82bc09e-a419-4d60-84bf-95d631fd0ebb -p 22 user@nas.lan cp -aRl full/snapshot/app_c74efccf-d273-46c9-8afe-3fd427bb78c1 full/2026-02-13-210356-064/app_git.jadud.com_v1.37.4 errored BoxError: ssh exited with code 1 signal null
Feb 13 16:11:00 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:82:23)
Feb 13 16:11:00 at ChildProcess.emit (node:events:519:28)
Feb 13 16:11:00 at maybeClose (node:internal/child_process:1101:16)
Feb 13 16:11:00 at ChildProcess._handle.onexit (node:internal/child_process:304:5) {
Feb 13 16:11:00 reason: 'Shell Error',
Feb 13 16:11:00 details: {},
Feb 13 16:11:00 stdout: <Buffer >,
Feb 13 16:11:00 stdoutString: '',
Feb 13 16:11:00 stdoutLineCount: 0,
Feb 13 16:11:00 stderr: <Buffer 63 70 3a 20 63 61 6e 6e 6f 74 20 73 74 61 74 20 27 66 75 6c 6c 2f 73 6e 61 70 73 68 6f 74 2f 61 70 70 5f 63 37 34 65 66 63 63 66 2d 64 32 37 33 2d 34 ... 50 more bytes>,
Feb 13 16:11:00 stderrString: "cp: cannot stat 'full/snapshot/app_c74efccf-d273-46c9-8afe-3fd427bb78c1': No such file or directory\n",
Feb 13 16:11:00 stderrLineCount: 1,
Feb 13 16:11:00 code: 1,
Feb 13 16:11:00 signal: null,
Feb 13 16:11:00 timedOut: false,
Feb 13 16:11:00 terminated: false
Feb 13 16:11:00 }
Feb 13 16:11:00 box:storage/filesystem SSH remote copy failed, trying sshfs copy
Feb 13 16:11:00 box:shell filesystem: cp -aRl /mnt/managedbackups/1ec6c6b4-7566-4369-b2ce-466968b00d5d/full/snapshot/app_c74efccf-d273-46c9-8afe-3fd427bb78c1 /mnt/managedbackups/1ec6c6b4-7566-4369-b2ce-466968b00d5d/full/2026-02-13-210356-064/app_git.jadud.com_v1.37.4
Feb 13 16:11:07 box:backuptask copy: copied successfully to 2026-02-13-210356-064/app_git.jadud.com_v1.37.4. Took 7.889 seconds