Backup fails Error finding empty dirs: spawnSync /bin/sh ENOBUFS

[archos]

Hi I have a problem with one server backup. The backup always fails with Error finding empty dirs: spawnSync /bin/sh ENOBUFS
He'll be grateful for any advice.

Feb 17 09:44:55box:services backupAddons: backing up ["localstorage","oidc","postgresql","sendmail","turn"]
Feb 17 09:44:56box:services pipeRequestToFile: connected with status code 200
Feb 17 10:13:00box:backuptask snapshotApp: matrix.mydomain
.cz took 1684.83 seconds
Feb 17 10:13:00box:tasks update 9051: {"percent":53.5,"message":"Uploading app snapshot matrix.mydomen.cz"}
Feb 17 10:13:00box:shell backup-snapshot/app_9afaff60-0e08-4215-8271-b19f8ff1540f spawn: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_9afaff60-0e08-4215-8271-b19f8ff1540f rsync {"localRoot":"/home/yellowtent/appsdata/9afaff60-0e08-4215-8271-b19f8ff1540f","layout":[]}
Feb 17 10:13:01box:shell backup-snapshot/app_9afaff60-0e08-4215-8271-b19f8ff1540f (stderr): 2024-02-17T09:13:01.748Z box:backupupload Backing up {"localRoot":"/home/yellowtent/appsdata/9afaff60-0e08-4215-8271-b19f8ff1540f","layout":[]} to snapshot/app_9afaff60-0e08-4215-8271-b19f8ff1540f
Feb 17 10:13:01box:shell backup-snapshot/app_9afaff60-0e08-4215-8271-b19f8ff1540f (stderr): 2024-02-17T09:13:01.834Z box:backuptask upload: path snapshot/app_9afaff60-0e08-4215-8271-b19f8ff1540f format rsync dataLayout {"localRoot":"/home/yellowtent/appsdata/9afaff60-0e08-4215-8271-b19f8ff1540f","layout":[]}
Feb 17 10:13:02box:shell backup-snapshot/app_9afaff60-0e08-4215-8271-b19f8ff1540f (stderr): 2024-02-17T09:13:02.325Z box:backuptask checkPreconditions: getting disk usage of /home/yellowtent/appsdata/9afaff60-0e08-4215-8271-b19f8ff1540f
Feb 17 10:13:02box:shell backup-snapshot/app_9afaff60-0e08-4215-8271-b19f8ff1540f (stderr): 2024-02-17T09:13:02.325Z box:backuptask upload: mount point status is {"state":"active"}
Feb 17 10:14:32box:shell backup-snapshot/app_9afaff60-0e08-4215-8271-b19f8ff1540f (stderr): 2024-02-17T09:14:32.957Z box:backuptask checkPreconditions: total required=166063174733 available=undefined
Feb 17 10:15:23box:shell backup-snapshot/app_9afaff60-0e08-4215-8271-b19f8ff1540f (stderr): 2024-02-17T09:15:23.890Z box:backupupload upload completed. error: BoxError: Error finding empty dirs: spawnSync /bin/sh ENOBUFS at saveFsMetadata (/home/yellowtent/box/src/backupformat/rsync.js:113:39) at Object.upload (/home/yellowtent/box/src/backupformat/rsync.js:250:11) at Object.upload (/home/yellowtent/box/src/backuptask.js:88:36) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) { reason: 'FileSystem Error', details: {} }
Feb 17 10:15:23box:backuptask runBackupUpload: result - {"result":"Error finding empty dirs: spawnSync /bin/sh ENOBUFS"}
Feb 17 10:15:23box:shell backup-snapshot/app_9afaff60-0e08-4215-8271-b19f8ff1540f code: 50, signal: null
Feb 17 10:15:23box:tasks setCompleted - 9051: {"result":null,"error":{"stack":"BoxError: Error finding empty dirs: spawnSync /bin/sh ENOBUFS\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:166:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"External Error","details":{},"message":"Error finding empty dirs: spawnSync /bin/sh ENOBUFS"}}
Feb 17 10:15:23box:tasks update 9051: {"percent":100,"result":null,"error":{"stack":"BoxError: Error finding empty dirs: spawnSync /bin/sh ENOBUFS\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:166:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"External Error","details":{},"message":"Error finding empty dirs: spawnSync /bin/sh ENOBUFS"}}
Feb 17 10:15:23box:taskworker Task took 7407.284 seconds
Error finding empty dirs: spawnSync /bin/sh ENOBUFS

[girish]

Oh, interesting. Maybe you have a lot of empty directories? Specifically, paths which are more than 30MB it seems.

Can you bump the '30' in the line at https://git.cloudron.io/cloudron/box/-/blob/master/src/backupformat/rsync.js?ref_type=heads#L112 ? Then try the backup again? Say, make it 100MB like below:

        const emptyDirs = safe.child_process.execSync(`find ${lp} -type d -empty`, { encoding: 'utf8', maxBuffer: 1024 * 1024 * 100 });

[archos]

@girish Thank you for your reply. Just please, can this be done in the backup settings, or do I need to set up a configuration file via SSH? Sorry for asking.
I have tried turning off the automatic backup of Matrix and the backup completes fine.

[girish]

You have to edit the source code. Specifically, the file at /home/yellowtent/box/src/backupformat/rsync.js .

Another idea: can you run this on the web terminal?

find /app/data -type d -empty | sed -e 's,/app/data/,/home/yellowtent/9afaff60-0e08-4215-8271-b19f8ff1540f/,' | wc -c

That should give us some idea about the size that the code needs.

[archos]

@girish The command in the web terminal didn't show me anything. I edited the code in /home/yellowtent/box/src/backupformat/rsync.js, so I'll try to run a backup, it will take a few hours. I have a lot of data.
I don't know if it's related, but lately Matrix has been putting a lot of strain on my CPU. I have removed about 30 Gb of media, but Matrix still has 50 Gb. There are about 150 users on there. VPS has 10 Core parameters "Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz"
Memory
63.22 GB RAM

[archos]

@girish Hi so the backup went fine after the code change. Thank you very much for your help.

[girish]

thanks, fixed in https://git.cloudron.io/cloudron/box/-/commit/1d1a7af48ebd41fa9a54f73c124a708c226e38e8