Nextcloud backup crashes

[jagan]

I have v7.5.2 (Ubuntu 20.04.5 LTS).
Backups are failing for me. I signed up for Backblaze B2 but the issue persists.
Signed up for a Hetzner CIFS storage box, but the CIFS did not mount for some reason.
Let me raise a support ticket with access to the server coz I don't know how to get the full backup logs.

[girish]

I am debugging @jagan 's setup on support. The core issue is the large amount of data being backed up (over 3TB). Disabling some of the apps which have this large data atleast makes the backup succeed. Trying to see what exactly makes the very large app which has over 1.5TB data fail.

[jagan]

Thanks Girish, I think I might have the largest Nextcloud on Cloudron. It has been working flawlessly, except for the present backup hiccup. I always feel so safe with Cloudron.
The support is priceless. As one of the earliest adopters, Cloudron has been my second-best decision ever.

[girish]

@jagan thanks for the nice words I think using a storage box is probably the best approach for backups instead of these object storage solutions. They are designed to serve reasonably sized objects and not 1.5TB objects You mentioned trying out Hetzner Storage Box but is your server on Hetzner? I don't have much idea how well CIFS works across the "internet"

[jagan]

I got a Storage Box on Hetzner, my Cloudron is also on Hetzner.
But somehow I am unable to add the storage box to the backup, may I please pass on the credentials to you please? if you can help me add it to the backup, I would be most grateful.
Will mail the details as a reply to the support mail.

[girish]

@jagan ah perfect, let's try that then. Is the hetzner storage and VPS in the same region ? I know that other network providers (like aws) also charge for outbound network! So, we are transferring out TB of data then there is a cost.

[nebulon]

Alternately you could also use the Hetzner storage box with sshfs instead of cifs. I have had quite a good experience with this.

[girish]

@jagan So, the mounting was failing because cifs-utils package was missing on the server. We added this in the install script a while ago, but I guess you are coming from a long time ago. Manually installing cifs-utils package makes it mount.

[jagan]

@girish said in Nextcloud backup crashes:

@jagan ah perfect, let's try that then. Is the hetzner storage and VPS in the same region ? I know that other network providers (like aws) also charge for outbound network! So, we are transferring out TB of data then there is a cost.

The VPS and Storage are in the same region of Germany.
There is sparse info on the storage.
I have to find out about the traffic. I hope internal traffic within Hetzner is free, but this is something I need to get sure about. Thanks for the pointer!

[girish]

I will mark this as solved for the moment. CIFS mounting works fine. The (still ongoing) issue is that the data is quite massive and I am looking into seeing if we can throw in some retry logic to make things more "stable".

[jagan]

@girish Yes Please. Thank you for the help.
I triggered a backup earlier today, and it stopped midway. Do the following log entries say anything of interest?

---

Dec 26 21:28:46 box:backuptask copy: copied to 2023-12-26-092602-586/app_hub.sbvu.ac.in_v4.20.5 errored. error: copy exited with code 1 signal null
Dec 26 21:28:47 box:apptask run: app error for state pending_backup: BoxError: copy exited with code 1 signal null at Object.copy (/home/yellowtent/box/src/storage/filesystem.js:191:26) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) { reason: 'External Error', details: {} }

---

[opensourced]

Sorry for not responding in an adequate time. Since my backups are still failing, I would try some other storage option. What storage protocol and configuration do you recommend for big backups?

[girish]

@opensourced I have been testing things on the side and tried copying over such amount of data to a CIFS mounted Hetzner Storage and just cp -r command failed in the middle. rclone failed as well since the Hetzner storage box mount point disconnects/breaks randomly. I am getting farther with rsync but currently cloudron does not have an adapter for rsync. We are still working on this. Fundamentally, backing up 2.5TB of data reliably is a very complicated task.

All this to say, I don't really have a solution but if someone has ideas, I can try something else.

[opensourced]

@girish Bevor I used Minio for backups, I was mounting a NFS mount. But I went away from using this, because there were instabilities with the mount and if the share was not mounted, the backup was made onto the rootfs and eventually filled up the disk untill the instance crashed.

[girish]

@opensourced er, apologies . I confused you and @jagan !

Can you tell a bit more about your network? Is it a home network or public VPS? Just trying to understand the practicalities of backing up so much. Where the minio and NFS in your local network ?

[eganonoa]

@girish said in Nextcloud backup crashes:

@opensourced I have been testing things on the side and tried copying over such amount of data to a CIFS mounted Hetzner Storage and just cp -r command failed in the middle. rclone failed as well since the Hetzner storage box mount point disconnects/breaks randomly. I am getting farther with rsync but currently cloudron does not have an adapter for rsync. We are still working on this. Fundamentally, backing up 2.5TB of data reliably is a very complicated task.

All this to say, I don't really have a solution but if someone has ideas, I can try something else.

@girish A few years ago you helped me out with a similar issue (not on forums but via a support ticket) with backing up a large Nextcloud instance (2TB+). The issue was with timeouts and you advised how and where to update to extend the timeout window significantly. I was using a Hetzner Storage Box in Germany and a VPS in Sweden using a CIFS mount. Later I moved it to an sshfs, rsync to a local server.

This fixed the issue and allowed me a couple of years of backups of that large Nextcloud instance (with many other apps running concurrently) without problem. I don't have those emails any more or access to that instance but wanted to mention in case a reminder is useful to you here as it seems to be a similar issue.

[opensourced]

@girish Cloudron is running in a Proxmox cluster. In the same local network there are storage servers via 10Gb/s. I guess ideal would be a NFS mount from a TrueNAS server. This way I could get rid of encryption overhead and use Jumbo frames. If cloudron could handle those NFS mounts stable, I would switch back to this. Other suggestions are very welcome.

[robi]

@girish can you make sure the remount checks for CIFS are applied to NFS as well? At every backup..

Having it be more bulletproof would be great!

[opensourced]

I would warmly welcome this and change right back to NFS.

[jagan]

Somehow, the backup is failing because files from the cache were not found.

Jan 17 06:05:54box:shell copy (stderr): /bin/cp: cannot create hard link '/mnt/cloudronbackup/2024-01-16-143004-490/app_n8n.sbvu.ac.in_v3.17.2/data/user/.cache/n8n/public/assets/@jsplumb/browser-ui-d5e5c8b4.js' to '/mnt/cloudronbackup/snapshot/app_3b812988-82fc-4270-b8b1-a94efa9fc06a/data/user/.cache/n8n/public/assets/@jsplumb/browser-ui-d5e5c8b4.js': No such file or directory