SSH remote copy always failed, falling back to sshfs copy

[RianKellyIT]

To be fair, the SSH remote copy failing is almost certainly a ZFS hardlink limitation rather than a permissions issue. Cloudron's backup system uses "cp -aRl" on the remote host to create a hardlink copy of the snapshot directory into a dated backup directory. This is the fast path that avoids re-uploading data between backup runs.

The problem with TrueNAS is that ZFS hardlinks only work within the same dataset. If your snapshot directory and your dated backup directories are on different ZFS datasets, or if the copy crosses a dataset boundary at any point, cp -aRl will fail silently (or with a "cross-device link" error) and Cloudron falls back to sshfs and uploads everything again.

A few things to check:

First, look at your TrueNAS dataset layout. If your Cloudron backup target is something like /mnt/pool/cloudron and the backup creates subdirectories inside it, everything needs to be within the same ZFS dataset. Run "zfs list" on TrueNAS and confirm that snapshot and the dated directories are under the same mount point.

Second, run the cp command manually to see the actual error. SSH in to TrueNAS as your backup user and try:
cp -aRl /path/to/snapshot /path/to/test-copy

If you see "cp: cannot create hard link: Invalid cross-device link" or similar, the dataset boundary is your problem.

Third, if you cannot fix the dataset layout, you can work around it by enabling Zstandard compression on the dataset and accepting the sshfs path, or by restructuring your TrueNAS datasets so the entire backup target is a single flat dataset with no child datasets underneath.

What does the log show right after the cp command fails? There should be an error message before it falls back to sshfs.

[Dummyzam]

Thanks for your great response!
I will check the the 3 points right now and report back.
But my TrueNas setup is very simple (it's my first TN install ). So i have only one zpol with several datasets in it. I don't use sub-dataset. So in almost all of my previous tests, the backup target was a dedicated dataset or a sub-directory of a dataset, with no cross-dataset reference (i didn't know it was possible ).
I did only one test with a sub-dataset dedicated to backup as target.
Every time with the same issue...
Is there a point i'm missing about your explanation ?
(and does the silent failure can explain the "cp: cannot stat No such file or directory" error in logs ?)

[Dummyzam]

Here the results :

1. zfs list a lot but here are the line corresponding to backup target (each used for different test, never at the same time)

admin@truenas[~]$ sudo zfs list
NAME                                                       USED  AVAIL  REFER  MOUNTPOINT
OnlyZpol                                                    649G  2.88T  6.30G  /mnt/OnlyZpol

OnlyZpol/CloudronBackup                                     256G  2.88T   256G  /mnt/OnlyZpol/CloudronBackup
[...]
OnlyZpol/UserTst                                            133G  2.88T  66.4G  /mnt/OnlyZpol/UserTst
OnlyZpol/UserTst/CloudronBck                               66.4G  2.88T  66.4G  /mnt/OnlyZpol/UserTst/CloudronBck
[...]
boot-pool                                                 28.4G  68.5G    96K  none

2. cp -aRl /path/to/snapshot /path/to/test-copy
   I did the test on both backup path i have, works and is quick.

myuser@truenas:~$ cp -aRl /mnt/OnlyZpol/UserTst/CloudronBck/snapshot/ /mnt/OnlyZpol/UserTst/CloudronBck/testdir/
myuser@truenas:~$ 
myuser@truenas:~$ cp -aRl /mnt/OnlyZpol/CloudronBackup/snapshot/ /mnt/OnlyZpol/CloudronBackup/testdir/
myuser@truenas:~$ 

3. Current compression is LZ4. I haven't change it to ZSTD yet as previous points. Moreover i'm not sure to understand the "accepting the sshfs path" part.

So, with my limited knowledge, i'm unsure my issue come from cross-dataset references.

But i will test again without hardlink.
EDIT : as expected, same issue, remote copy fail....

[joseph]

Have you tried running ssh -o "StrictHostKeyChecking no" -i /tmp/identity_file_dba91bb5-5fb9-46b8-91cc-69d69437aaea -p 22 myuser@192.168.0.10 cp -aRl snapshot/app_6da85c9a-fbf0-4563-baaa-1ad3080cf467 temp ?

[Dummyzam]

Thanks for pointing this because no, i didn't.
But now yes, and i have the same issue.
/tmp/identity_file... was not found but i was able to log with password (i need to remove that).
I did the test with an absolute path and it woks.
I'm not sure how i should interpret all of it... Is this an issue similar to this topic ? Have i set-up sshfs target incorrectly causing the relative path to fail ? Is this another connection/permission issue as i saw on another post i can't find ?

[jadudm]

Is Cloudron using a correct/full path when issuing the cp over ssh? I've had this problem, too, and I know I'm not spanning ZFS pools. If we enter a root path into the config, should't Cloudron use the full paths, for correctness/completeness/clarity, when issuing the cp?

I don't think that helps much, but I've seen the same persistent issues that @dummyzam is pointing to.

[joseph]

yeah, it's always absolute paths.

[Dummyzam]

thanks @joseph for your answer.
But the command you gave me (same one as in error log) use relative path.
And when i try the same command manually with an absolute path, it seems to works.
What should i understand of it ?

[joseph]

@Dummyzam you are right. For sshfs, the paths are relative (to the user's home). The cp command is run after ssh into that server.

I did the test with an absolute path and it woks.

What did you mean by this? I think we have to figure out why ssh -p 22 myuser@192.168.0.10 cp -aRl snapshot/app_6da85c9a-fbf0-4563-baaa-1ad3080cf467 temp does not work. Is that because the ssh user has some other home directory? Maybe your prefix (in the backup site config) is an absolute path and not a relative path?

[Dummyzam]

@joseph said:

he paths are relative (to the user's home)

Is it relative to user $home or to target directory ?
Because if it is to the target directory, it should have been '../CloudronBackup/snapshot' in my case and not 'snapshot'

Because

@joseph said:
What did you mean by this?

My user $home is : /mnt/OnlyZpol/UserTst/
My target directory is : /mnt/OnlyZpol/CloudronBackup/

Running :

ssh -o "StrictHostKeyChecking no" -p 22 myuser@192.168.0.10 cp -aRl snapshot/app_6da85c9a-fbf0-4563-baaa-1ad3080cf467 temp

-> FAIL

ssh -o "StrictHostKeyChecking no" -p 22 myuser@192.168.0.10 cp -aRl /mnt/OnlyZpol/CloudronBackup/snapshot/app_6da85c9a-fbf0-4563-baaa-1ad3080cf467 /mnt/OnlyZpol/CloudronBackup/temp

-> Success

Is that because the ssh user has some other home directory? Maybe your prefix (in the backup site config) is an absolute path and not a relative path?

So yes for the first and no for the second, i never set an "absolute prefix".
I will try to use a subdirectory of user's home, but i've already tried quite similar with a sub-dataset ; without luck.

IMO it whould be safer to give absolute path in sshfs as you can't really ensure ssh will login each time in same directory

[joseph]

@Dummyzam I think you can set the prefix to /mnt/OnlyZpol/CloudronBackup . That will make cp use absolute paths.