Backup fails Error finding empty dirs: spawnSync /bin/sh ENOBUFS
-
Hi I have a problem with one server backup. The backup always fails with Error finding empty dirs: spawnSync /bin/sh ENOBUFS
He'll be grateful for any advice.Feb 17 09:44:55box:services backupAddons: backing up ["localstorage","oidc","postgresql","sendmail","turn"] Feb 17 09:44:56box:services pipeRequestToFile: connected with status code 200 Feb 17 10:13:00box:backuptask snapshotApp: matrix.mydomain .cz took 1684.83 seconds Feb 17 10:13:00box:tasks update 9051: {"percent":53.5,"message":"Uploading app snapshot matrix.mydomen.cz"} Feb 17 10:13:00box:shell backup-snapshot/app_9afaff60-0e08-4215-8271-b19f8ff1540f spawn: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_9afaff60-0e08-4215-8271-b19f8ff1540f rsync {"localRoot":"/home/yellowtent/appsdata/9afaff60-0e08-4215-8271-b19f8ff1540f","layout":[]} Feb 17 10:13:01box:shell backup-snapshot/app_9afaff60-0e08-4215-8271-b19f8ff1540f (stderr): 2024-02-17T09:13:01.748Z box:backupupload Backing up {"localRoot":"/home/yellowtent/appsdata/9afaff60-0e08-4215-8271-b19f8ff1540f","layout":[]} to snapshot/app_9afaff60-0e08-4215-8271-b19f8ff1540f Feb 17 10:13:01box:shell backup-snapshot/app_9afaff60-0e08-4215-8271-b19f8ff1540f (stderr): 2024-02-17T09:13:01.834Z box:backuptask upload: path snapshot/app_9afaff60-0e08-4215-8271-b19f8ff1540f format rsync dataLayout {"localRoot":"/home/yellowtent/appsdata/9afaff60-0e08-4215-8271-b19f8ff1540f","layout":[]} Feb 17 10:13:02box:shell backup-snapshot/app_9afaff60-0e08-4215-8271-b19f8ff1540f (stderr): 2024-02-17T09:13:02.325Z box:backuptask checkPreconditions: getting disk usage of /home/yellowtent/appsdata/9afaff60-0e08-4215-8271-b19f8ff1540f Feb 17 10:13:02box:shell backup-snapshot/app_9afaff60-0e08-4215-8271-b19f8ff1540f (stderr): 2024-02-17T09:13:02.325Z box:backuptask upload: mount point status is {"state":"active"} Feb 17 10:14:32box:shell backup-snapshot/app_9afaff60-0e08-4215-8271-b19f8ff1540f (stderr): 2024-02-17T09:14:32.957Z box:backuptask checkPreconditions: total required=166063174733 available=undefined Feb 17 10:15:23box:shell backup-snapshot/app_9afaff60-0e08-4215-8271-b19f8ff1540f (stderr): 2024-02-17T09:15:23.890Z box:backupupload upload completed. error: BoxError: Error finding empty dirs: spawnSync /bin/sh ENOBUFS at saveFsMetadata (/home/yellowtent/box/src/backupformat/rsync.js:113:39) at Object.upload (/home/yellowtent/box/src/backupformat/rsync.js:250:11) at Object.upload (/home/yellowtent/box/src/backuptask.js:88:36) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) { reason: 'FileSystem Error', details: {} } Feb 17 10:15:23box:backuptask runBackupUpload: result - {"result":"Error finding empty dirs: spawnSync /bin/sh ENOBUFS"} Feb 17 10:15:23box:shell backup-snapshot/app_9afaff60-0e08-4215-8271-b19f8ff1540f code: 50, signal: null Feb 17 10:15:23box:tasks setCompleted - 9051: {"result":null,"error":{"stack":"BoxError: Error finding empty dirs: spawnSync /bin/sh ENOBUFS\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:166:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"External Error","details":{},"message":"Error finding empty dirs: spawnSync /bin/sh ENOBUFS"}} Feb 17 10:15:23box:tasks update 9051: {"percent":100,"result":null,"error":{"stack":"BoxError: Error finding empty dirs: spawnSync /bin/sh ENOBUFS\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:166:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"External Error","details":{},"message":"Error finding empty dirs: spawnSync /bin/sh ENOBUFS"}} Feb 17 10:15:23box:taskworker Task took 7407.284 seconds Error finding empty dirs: spawnSync /bin/sh ENOBUFS
-
Oh, interesting. Maybe you have a lot of empty directories? Specifically, paths which are more than 30MB it seems.
Can you bump the '30' in the line at https://git.cloudron.io/cloudron/box/-/blob/master/src/backupformat/rsync.js?ref_type=heads#L112 ? Then try the backup again? Say, make it 100MB like below:
const emptyDirs = safe.child_process.execSync(`find ${lp} -type d -empty`, { encoding: 'utf8', maxBuffer: 1024 * 1024 * 100 });
-
-
-
You have to edit the source code. Specifically, the file at
/home/yellowtent/box/src/backupformat/rsync.js
.Another idea: can you run this on the web terminal?
find /app/data -type d -empty | sed -e 's,/app/data/,/home/yellowtent/9afaff60-0e08-4215-8271-b19f8ff1540f/,' | wc -c
That should give us some idea about the size that the code needs.
-
@girish The command in the web terminal didn't show me anything. I edited the code in /home/yellowtent/box/src/backupformat/rsync.js, so I'll try to run a backup, it will take a few hours. I have a lot of data.
I don't know if it's related, but lately Matrix has been putting a lot of strain on my CPU. I have removed about 30 Gb of media, but Matrix still has 50 Gb. There are about 150 users on there. VPS has 10 Core parameters "Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz"
Memory
63.22 GB RAM -