The NFS disconnection issue or if a mountpoint otherwise goes bad is indeed still an issue though.

@girish if i knew that there is some NFS improve scheduled, i could work around untill then. If not, i would need to cope with the current circumstances somehow. I think there would not even much needed.

I just need the NFS connector to:

• Never write to the root mount in order to prevent full root disk.
• Check mount before backup.
• If during the backup there is a connection issue, try to remount and continue. Try remount for 10 times with a delay of 10s or so...

@girish can you make sure the remount checks for CIFS are applied to NFS as well? At every backup..

Having it be more bulletproof would be great!

@girish do you think there will be a more stable support for NFS in the feature?

Jan 17 06:05:54box:shell copy (stderr): /bin/cp: cannot create hard link '/mnt/cloudronbackup/2024-01-16-143004-490/app_n8n.sbvu.ac.in_v3.17.2/data/user/.cache/n8n/public/assets/@jsplumb/browser-ui-d5e5c8b4.js' to '/mnt/cloudronbackup/snapshot/app_3b812988-82fc-4270-b8b1-a94efa9fc06a/data/user/.cache/n8n/public/assets/@jsplumb/browser-ui-d5e5c8b4.js': No such file or directory

Having it be more bulletproof would be great!

@opensourced I have been testing things on the side and tried copying over such amount of data to a CIFS mounted Hetzner Storage and just cp -r command failed in the middle. rclone failed as well since the Hetzner storage box mount point disconnects/breaks randomly. I am getting farther with rsync but currently cloudron does not have an adapter for rsync. We are still working on this. Fundamentally, backing up 2.5TB of data reliably is a very complicated task.

All this to say, I don't really have a solution but if someone has ideas, I can try something else.

@girish A few years ago you helped me out with a similar issue (not on forums but via a support ticket) with backing up a large Nextcloud instance (2TB+). The issue was with timeouts and you advised how and where to update to extend the timeout window significantly. I was using a Hetzner Storage Box in Germany and a VPS in Sweden using a CIFS mount. Later I moved it to an sshfs, rsync to a local server.

This fixed the issue and allowed me a couple of years of backups of that large Nextcloud instance (with many other apps running concurrently) without problem. I don't have those emails any more or access to that instance but wanted to mention in case a reminder is useful to you here as it seems to be a similar issue.

Can you tell a bit more about your network? Is it a home network or public VPS? Just trying to understand the practicalities of backing up so much. Where the minio and NFS in your local network ?

All this to say, I don't really have a solution but if someone has ideas, I can try something else.

Dec 26 21:28:46 box:backuptask copy: copied to 2023-12-26-092602-586/app_hub.sbvu.ac.in_v4.20.5 errored. error: copy exited with code 1 signal null
Dec 26 21:28:47 box:apptask run: app error for state pending_backup: BoxError: copy exited with code 1 signal null at Object.copy (/home/yellowtent/box/src/storage/filesystem.js:191:26) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) { reason: 'External Error', details: {} }

@jagan ah perfect, let's try that then. Is the hetzner storage and VPS in the same region ? I know that other network providers (like aws) also charge for outbound network! So, we are transferring out TB of data then there is a cost.

The VPS and Storage are in the same region of Germany.
There is sparse info on the storage.
I have to find out about the traffic. I hope internal traffic within Hetzner is free, but this is something I need to get sure about. Thanks for the pointer!