Backup error 'copy exited with code 1 signal null'

[martinkbs]

Hi guys

In the last two days I am getting the following 'Failed to backup' email with the following text:

Cloudron failed to create a complete backup. Please see https://docs.cloudron.io/troubleshooting/#backups for troubleshooting.

Checking the backup system, I verify that the backup has been generated in all the applications, but in the Cloudron backups section the error 'copy exited with code 1 signal null' appears.

Reviewing the log, these are the last lines that show where the failure occurs:

Mar 24 11:58:36 box:tasks 356: {"percent":97.875,"message":"box: Copying /mnt/backups/snapshot/box.tar.gz to /mnt/backups/2021-03-24-101056-946/box_2021-03-24-105836-281_v6.2.4.tar.gz"}
Mar 24 11:58:36 box:shell copy spawn: /bin/cp -a /mnt/backups/snapshot/box.tar.gz /mnt/backups/2021-03-24-101056-946/box_2021-03-24-105836-281_v6.2.4.tar.gz
Mar 24 11:58:46 box:shell copy (stdout): /bin/cp: error reading '/mnt/backups/snapshot/box.tar.gz'
Mar 24 11:58:46 box:shell copy (stdout): : Input/output error
Mar 24 11:58:46 box:shell copy code: 1, signal: null
Mar 24 11:58:46 box:taskworker Task took 2869.561 seconds
Mar 24 11:58:46 box:tasks setCompleted - 356: {"result":null,"error":{"stack":"BoxError: copy exited with code 1 signal null\n at /home/yellowtent/box/src/storage/filesystem.js:203:51\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:315:20)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)","name":"BoxError","reason":"External Error","details":{},"message":"copy exited with code 1 signal null"}}
Mar 24 11:58:46 box:tasks 356: {"percent":100,"result":null,"error":{"stack":"BoxError: copy exited with code 1 signal null\n at /home/yellowtent/box/src/storage/filesystem.js:203:51\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:315:20)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)","name":"BoxError","reason":"External Error","details":{},"message":"copy exited with code 1 signal null"}}

Apparently, the failure occurs when the Cloudron itself is backed up, since as I said before, the copies of all the applications are made.

Any hints or suggestions to solve the problem?

Thanks.

[nebulon]

Is it possible that the backup location is some mounted network filesystem or so?
What is failing looks like a regular file copy, so the I/O error indicates a filesystem problem here. Maybe try to remount this?

[martinkbs]

The backup system configuration is as follows:

The folder has a Google Drive team drive mounted with rClone to store the backups.

It has been working normally from the beginning except for the last two days, which is when it has started to give this error.

Cloudron is already updated to version 6.2.4

[nebulon]

@martinkbs have you tried to remount this and trigger a new backup?

[martinkbs]

@nebulon Yes, the drive mounts automatically every time the server is rebooted via crontab.

I have rebooted the server, removed several heavy applications from the backup (to rule out memory issues) and the only LAMP application containing non-Cloudron code.

The result of the new backup is the same, the error occurs when saving the file /mnt/backups/snapshot/box.tar.gz

Mar 24 16:11:15 box:shell backup-snapshot/box spawn: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/box tgz {"localRoot":"/home/yellowtent/boxdata","layout":[]}
Mar 24 16:11:16 box:shell backup-snapshot/box (stdout): 2021-03-24T15:11:16.764Z box:backupupload Backing up {"localRoot":"/home/yellowtent/boxdata","layout":[]} to snapshot/box
Mar 24 16:11:16 box:shell backup-snapshot/box (stdout): 2021-03-24T15:11:16.817Z box:settings initCache: pre-load settings
Mar 24 16:11:16 box:shell backup-snapshot/box (stdout): 2021-03-24T15:11:16.834Z box:backupupload process: rss: 73887744 heapTotal: 55631872 heapUsed: 22185072 external: 1840481
Mar 24 16:11:16 box:shell backup-snapshot/box (stdout): 2021-03-24T15:11:16.834Z box:backupupload v8 heap : used 22189400 total: 55631872 max: 4345298944
Mar 24 16:11:16 box:backups upload: id snapshot/box format tgz dataLayout {"localRoot":"/home/yellowtent/boxdata","layout":[]}
Mar 24 16:11:16 box:shell backup-snapshot/box (stdout): 2021-03-24T15:11:16.847Z box:storage/filesystem checkPreconditions: getting disk usage of /home/yellowtent/boxdata
Mar 24 16:11:16 box:shell backup-snapshot/box (stdout): 2021-03-24T15:11:16.859Z box:storage/filesystem checkPreconditions: 3872786 bytes
Mar 24 16:11:18 box:tasks 359: {"percent":97.875,"message":"Uploading backup 1M@1MBps (box)"}
Mar 24 16:11:18 box:shell backup-snapshot/box (stdout): 2021-03-24T15:11:18.267Z box:storage/filesystem upload /mnt/backups/snapshot/box.tar.gz: done.
Mar 24 16:11:18 box:shell backup-snapshot/box (stdout): 2021-03-24T15:11:18.267Z box:backupupload upload completed. error: null
Mar 24 16:11:18 box:backups runBackupUpload: result - {"result":""}
Mar 24 16:11:19 box:backups uploadBoxSnapshot: took 3.414 seconds
Mar 24 16:11:19 box:backups Rotating box backup to id 2021-03-24-145230-064/box_2021-03-24-151119-388_v6.2.4
Mar 24 16:11:19 box:tasks 359: {"percent":97.875,"message":"box: Copying /mnt/backups/snapshot/box.tar.gz to /mnt/backups/2021-03-24-145230-064/box_2021-03-24-151119-388_v6.2.4.tar.gz"}
Mar 24 16:11:19 box:shell copy spawn: /bin/cp -a /mnt/backups/snapshot/box.tar.gz /mnt/backups/2021-03-24-145230-064/box_2021-03-24-151119-388_v6.2.4.tar.gz
Mar 24 16:11:37 box:shell copy (stdout): /bin/cp: error reading '/mnt/backups/snapshot/box.tar.gz': Input/output error
Mar 24 16:11:37 box:shell copy code: 1, signal: null
Mar 24 16:11:37 box:taskworker Task took 1147.049 seconds
Mar 24 16:11:37 box:tasks setCompleted - 359: {"result":null,"error":{"stack":"BoxError: copy exited with code 1 signal null\n at /home/yellowtent/box/src/storage/filesystem.js:203:51\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:315:20)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)","name":"BoxError","reason":"External Error","details":{},"message":"copy exited with code 1 signal null"}}
Mar 24 16:11:37 box:tasks 359: {"percent":100,"result":null,"error":{"stack":"BoxError: copy exited with code 1 signal null\n at /home/yellowtent/box/src/storage/filesystem.js:203:51\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:315:20)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)","name":"BoxError","reason":"External Error","details":{},"message":"copy exited with code 1 signal null"}}

Is this file the only one to be replaced? Is that so?

Could it be that when it is replaced (and a new one is not generated like the rest of the application files) it may give an error due to the time lag in the synchronization with Google Drive through rClone?

What I do not know is why this error has started to occur for two days, when the copy system has worked well in the previous days ...

[nebulon]

@martinkbs it could very much be a timing issue or maybe even some network connection speed change to your google drive. Is there by any chance some way to mount the drive with some sync flag? Might reduce overall speed but maybe is more solid. I don't have any experience with rclone mounts from google drive though.

[rareskyfive]

Click on the file path located in the Host Path column and a window will open allowing you to locate the active file. If you do not need to back up this file then you can edit the backup plan removing that file from your backup.

[martinkbs]

In the rClone documentation I see that there is the following flag that may solve the writing problem of box.tar.gz:

--delete-before          When synchronizing, delete files on destination before transferring

I will change the mount command within the crontab and launch a new test to see if it is corrected that way.

Thanks for the help @nebulon

[robi]

@martinkbs it will be good to have documented the tested rclone setup as part of the documentation!