Backup Fails: "Unknown system error -74"

[rlp10]

I am getting the following error each time I try to backup:

"Unknown system error -74: Unknown system error -74, open ..." and then the path to my backup.

I am using backup over sshfs. At first it seemed to work fine, but now it reliably fails with this same error.

If I watch the backup, then it seems to fail during the Nextcloud phase of the backup and I wonder if the problems relates to Nextcloud.

I have tried searching for that error number, but I can hardly find anything. My only hit is that it is possibly an error generating by nodejs. Perhaps node is being used to perform the backup or elsewhere in the Cloudron infrastructure?

I can SSH into the cloudron server and browse the remote server by going to /backups_sshfs/. I can view the backups there and it seems to work relatively promptly.

Otherwise, I'm out of ideas. Does anyone have any suggestions for how to troubleshoot this issue?

[nebulon]

Did you try to remount that volume and retrigger the backup?
Otherwise do you know if it fails consistently at the same file and also are you using backup encryption?

[rlp10]

@nebulon Thanks for your reply and sorry for the delay - I do still have the same issue.

I avoided the error by using tarballs to backup rather than rsync. However, I now want to sort it out properly with rsync and hard links as the space taken by tarballs is just too much.

I ran the backup again yesterday and I still have the same error message.

To answer your questions:

I have remounted and retriggered the backup on several occasions - it's always the same error.

I am using encryption, yes.

It does always seem to be the same application that causes the problem. Looking at the ID and browsing the /home/yellowtent directory on my server, I think the UUID refers to my Nextcloud installation. Admittedly, that would be the app with the most data (by far).

Perhaps I could try running the backup with Nextcloud offline to see if that works. It wouldn't be a permanent solution, but it would be interesting to know if that works at least.

Also, perhaps I should be setting up a separate "volume" for my Nextcloud data and migrate it over. I am a little nervous about that though because I'd have to make sure that all the sharing links etc in Nextcloud still worked.

Any other ideas how to troubleshoot this? Thanks in advance.

[nebulon]

@rlp10 that is useful to know already. You could just trigger a backup of that Nextcloud instance alone and it should result in the same error. If so, can you maybe share the app logs around the area of the error?

If you don't want to reveal file paths here publicly, you can also send a mail with the logs attached to support@cloudron.io

[rlp10]

@nebulon Thanks for that.

Yes it does seem to fail even when I just backup Nextcloud.

Looking at the logs, it appears to be the same directory that is always failing. Inspecting that directory, it seems to have very long filenames.

I've therefore compressed the directory into a tarball and then deleted the original. I'm re-running the backup to see if it will work now. Perhaps I'm hitting up against a limit on the length of filenames.

If it fails, then I will let you have the logs as you have kindly suggested.

[nebulon]

@rlp10 then this is most likely due to the long file name + encryption. Filepaths can only be so long as the target filesystem/storage supports and usually this is about 256 characters. The filepath encryption adds a few more characters to the path. In you case you could also turn off encryption if that is feasible for you.

[rlp10]

@nebulon Ugh, it still didn't work despite my having removed the previous directory where it failed. It is just failing somewhere else now.

Following your latest post, I'm going to try a backup which is not encrypted to see if that works.

Failing that, do you think it might be more reliable if I were to backup to a Minio installation, rather than using sshfs? I'm running a Ubuntu-based distro on the machine holding the backups, so it wouldn't be very difficult to install, I imagine.

I could even try it out in docker first, just to see if it fixed it.

[girish]

@rlp10 You can see the filename length limitations here - https://docs.cloudron.io/backups/#encryption

[rlp10]

@girish Thanks for that information

I'm pleased to say that the backup did run without encryption. Since I own the target backup machine, I will just proceed with that.

I have to admit, I'm relieved to see a complete backup of my data!

If anyone does get the chance, I would be grateful to know if Minio is preferred as a backup target in comparison with sshfs?

Also thanks @nebulon for all your help troubleshooting this issue.

[nebulon]

@rlp10 there is no clear cut between which backup target is preferred. We try to cover a few options to cover a range or use-cases, but all should work.

If you own the backup machine, I would anyways not advise to encrypt backups. Unencrypted files on trusted storage is a lot easier to restore if worst case trouble hits