Nextcloud backup crashes

opensourced

I have a Nextcloud running on one of my instances, that uses a hd-storage mount of around 2.5 TB size.

The problem is, that instance does not manage to finish the backup. The machine has 32GB memory and 4 cores. The Nextcloud App has a memory limit of 8GB and CPU shares of 50%.

The following is the backup configuration.

The Minio instance is a self hosted app that runs on another Cloudron.

When the backup failed I saw this log:

Nov 19 18:52:17box:tasks update 8098: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)\n at async Object.backupApp (/home/yellowtent/box/src/backuptask.js:321:12)\n at async backup (/home/yellowtent/box/src/apptask.js:363:22)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
Backuptask crashed
at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)
at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)
at async Object.backupApp (/home/yellowtent/box/src/backuptask.js:321:12)
at async backup (/home/yellowtent/box/src/apptask.js:363:22)

The issue might be related with timeouts, because the Minio is not the fastest instance. Nevertheless, I would expect this to work, since its not really a huge amount of data.

Thanks for any suggestions or information on how to further debug this issue.

robi

Try bumping memory for the backup config in the advanced section.

nebulon

Maybe the concurrency could be lower to not overflow the system also with I/O

jagan

I got the same issue.
Here are my logs:

Dec 02 06:27:28box:shell backup-snapshot/app_00ca9de5-a6cc-45b8-8914-f23a6609a145 code: null, signal: SIGKILL
Dec 02 06:27:28box:taskworker Task took 32248.202 seconds
Dec 02 06:27:28box:tasks setCompleted - 16719: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
Dec 02 06:27:28box:tasks update 16719: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
Backuptask crashed

And my backup settings are as follows:

Should I increase the memory? Let me bump it to 2 GB.
Looks like Upload Concurrency of 10 is the minimum.
I just changed from Wasabi (the price bump was too much) to iDrive e2.
Thanks for any help and pointers.

girish

@opensourced @jagan It says backup crashed. This indicates some real crash in code. Can you send me the full backup logs to support@cloudron.io ?

Is there a chance you both are in Ubuntu 18 (check with lsb_release -a or the Settings view)? if so, this could be related to be node version mismatch issue.

jagan

I have v7.5.2 (Ubuntu 20.04.5 LTS).
Backups are failing for me. I signed up for Backblaze B2 but the issue persists.
Signed up for a Hetzner CIFS storage box, but the CIFS did not mount for some reason.
Let me raise a support ticket with access to the server coz I don't know how to get the full backup logs.

girish

I am debugging @jagan 's setup on support. The core issue is the large amount of data being backed up (over 3TB). Disabling some of the apps which have this large data atleast makes the backup succeed. Trying to see what exactly makes the very large app which has over 1.5TB data fail.

jagan

Thanks Girish, I think I might have the largest Nextcloud on Cloudron. It has been working flawlessly, except for the present backup hiccup. I always feel so safe with Cloudron.
The support is priceless. As one of the earliest adopters, Cloudron has been my second-best decision ever.

girish

@jagan thanks for the nice words I think using a storage box is probably the best approach for backups instead of these object storage solutions. They are designed to serve reasonably sized objects and not 1.5TB objects You mentioned trying out Hetzner Storage Box but is your server on Hetzner? I don't have much idea how well CIFS works across the "internet"

jagan

I got a Storage Box on Hetzner, my Cloudron is also on Hetzner.
But somehow I am unable to add the storage box to the backup, may I please pass on the credentials to you please? if you can help me add it to the backup, I would be most grateful.
Will mail the details as a reply to the support mail.

girish

@jagan ah perfect, let's try that then. Is the hetzner storage and VPS in the same region ? I know that other network providers (like aws) also charge for outbound network! So, we are transferring out TB of data then there is a cost.

nebulon

Alternately you could also use the Hetzner storage box with sshfs instead of cifs. I have had quite a good experience with this.

girish

@jagan So, the mounting was failing because cifs-utils package was missing on the server. We added this in the install script a while ago, but I guess you are coming from a long time ago. Manually installing cifs-utils package makes it mount.

jagan

@girish said in Nextcloud backup crashes:

@jagan ah perfect, let's try that then. Is the hetzner storage and VPS in the same region ? I know that other network providers (like aws) also charge for outbound network! So, we are transferring out TB of data then there is a cost.

The VPS and Storage are in the same region of Germany.
There is sparse info on the storage.
I have to find out about the traffic. I hope internal traffic within Hetzner is free, but this is something I need to get sure about. Thanks for the pointer!

girish

I will mark this as solved for the moment. CIFS mounting works fine. The (still ongoing) issue is that the data is quite massive and I am looking into seeing if we can throw in some retry logic to make things more "stable".

jagan

@girish Yes Please. Thank you for the help.
I triggered a backup earlier today, and it stopped midway. Do the following log entries say anything of interest?

Dec 26 21:28:46 box:backuptask copy: copied to 2023-12-26-092602-586/app_hub.sbvu.ac.in_v4.20.5 errored. error: copy exited with code 1 signal null
Dec 26 21:28:47 box:apptask run: app error for state pending_backup: BoxError: copy exited with code 1 signal null at Object.copy (/home/yellowtent/box/src/storage/filesystem.js:191:26) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) { reason: 'External Error', details: {} }

opensourced

Sorry for not responding in an adequate time. Since my backups are still failing, I would try some other storage option. What storage protocol and configuration do you recommend for big backups?

girish

@opensourced I have been testing things on the side and tried copying over such amount of data to a CIFS mounted Hetzner Storage and just cp -r command failed in the middle. rclone failed as well since the Hetzner storage box mount point disconnects/breaks randomly. I am getting farther with rsync but currently cloudron does not have an adapter for rsync. We are still working on this. Fundamentally, backing up 2.5TB of data reliably is a very complicated task.

All this to say, I don't really have a solution but if someone has ideas, I can try something else.

opensourced

@girish Bevor I used Minio for backups, I was mounting a NFS mount. But I went away from using this, because there were instabilities with the mount and if the share was not mounted, the backup was made onto the rootfs and eventually filled up the disk untill the instance crashed.

girish

@opensourced er, apologies . I confused you and @jagan !

Can you tell a bit more about your network? Is it a home network or public VPS? Just trying to understand the practicalities of backing up so much. Where the minio and NFS in your local network ?

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

Nextcloud backup crashes