Since update to v8.2.1 backups fail with "Too many empty directories"

[fbartels]

Hi,

this morning I noticed that since the 31st (and the update to 8.2.1 from 8.1.0) my backups keep failing with the same error message:

Too many empty directories. Run "find /home/yellowtent/boxdata/mail -type d -empty" to investigate

In the log this can be found for such a task:

2025-01-03T07:15:34.868Z box:shell backupformat/rsync: find /home/yellowtent/boxdata/mail -type d -empty
2025-01-03T07:15:41.125Z box:shell backupformat/rsync: find /home/yellowtent/boxdata/mail -type d -empty errored BoxError: find exited with code null signal SIGKILL
    at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:72:23)
    at ChildProcess.emit (node:events:519:28)
    at maybeClose (node:internal/child_process:1105:16)
    at Socket.<anonymous> (node:internal/child_process:457:11)
    at Socket.emit (node:events:519:28)
    at Pipe.<anonymous> (node:net:339:12) {
  reason: 'Shell Error',
  details: {},
  stdout: '/home/yellowtent/boxdata/mail/solr\n' +
[..]
    '/home/yellowtent/boxdata'... 2599152 more characters,
  stdoutLineCount: 20073,
  stderr: '',
  stderrLineCount: 0,
  code: null,
  signal: 'SIGKILL'
}
2025-01-03T07:15:41.205Z box:backupupload upload completed. error: BoxError: Too many empty directories. Run "find /home/yellowtent/boxdata/mail -type d -empty" to investigate at saveFsMetadata (/home/yellowtent/box/src/backupformat/rsync.js:131:82) at async Object.upload (/home/yellowtent/box/src/backupformat/rsync.js:270:5) at async Object.upload (/home/yellowtent/box/src/backuptask.js:92:5) { reason: 'FileSystem Error', details: {} }
2025-01-03T07:15:41.373Z box:backuptask runBackupUpload: result - {"result":"Too many empty directories. Run \"find /home/yellowtent/boxdata/mail -type d -empty\" to investigate"}
2025-01-03T07:15:41.533Z box:shell backuptask: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/mail rsync {"localRoot":"/home/yellowtent/boxdata/mail","layout":[]} errored BoxError: backuptask exited with code 50 signal null
    at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19)
    at ChildProcess.emit (node:events:519:28)
    at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
  reason: 'Shell Error',
  details: {},
  code: 50,
  signal: null
}
2025-01-03T07:15:41.612Z box:taskworker Task took 415.137 seconds
2025-01-03T07:15:41.614Z box:tasks setCompleted - 19010: {"result":null,"error":{"stack":"BoxError: Too many empty directories. Run \"find /home/yellowtent/boxdata/mail -type d -empty\" to investigate\n    at runBackupUpload (/home/yellowtent/box/src/backuptask.js:172:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async uploadMailSnapshot (/home/yellowtent/box/src/backuptask.js:413:5)\n    at async backupMailWithTag (/home/yellowtent/box/src/backuptask.js:462:5)\n    at async fullBackup (/home/yellowtent/box/src/backuptask.js:523:26)","name":"BoxError","reason":"External Error","details":{},"message":"Too many empty directories. Run \"find /home/yellowtent/boxdata/mail -type d -empty\" to investigate"}}
2025-01-03T07:15:41.615Z box:tasks update 19010: {"percent":100,"result":null,"error":{"stack":"BoxError: Too many empty directories. Run \"find /home/yellowtent/boxdata/mail -type d -empty\" to investigate\n    at runBackupUpload (/home/yellowtent/box/src/backuptask.js:172:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async uploadMailSnapshot (/home/yellowtent/box/src/backuptask.js:413:5)\n    at async backupMailWithTag (/home/yellowtent/box/src/backuptask.js:462:5)\n    at async fullBackup (/home/yellowtent/box/src/backuptask.js:523:26)","name":"BoxError","reason":"External Error","details":{},"message":"Too many empty directories. Run \"find /home/yellowtent/boxdata/mail -type d -empty\" to investigate"}}
BoxError: Too many empty directories. Run "find /home/yellowtent/boxdata/mail -type d -empty" to investigate
    at runBackupUpload (/home/yellowtent/box/src/backuptask.js:172:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async uploadMailSnapshot (/home/yellowtent/box/src/backuptask.js:413:5)
    at async backupMailWithTag (/home/yellowtent/box/src/backuptask.js:462:5)
    at async fullBackup (/home/yellowtent/box/src/backuptask.js:523:26)

What I am wondering about is the find exited with code null signal SIGKILL part. null seems a weird exit code and also running the find command manually finishes with exit code 0. As far as I can see the server should not be running out of resources during this and no related error seems to be in dmesg.

[girish]

@fbartels this comes from https://git.cloudron.io/platform/box/-/blob/master/src/backupformat/rsync.js?ref_type=heads#L130 . The command it runs is find /home/yellowtent/boxdata/mail -type d -empty and somehow this exceeds 20000 lines (in stdout or stderr). Can you check why by running it manually?

[fbartels]

I have a testing mailbox called "many-folders", which on its own already exceeds this limit:

find /home/yellowtent/boxdata/mail -type d -empty | grep many-folders | wc -l
23811

The total line count when not grepping for my test mailbox is 31099.

Depending on the amount of users and if these users are folder hoarders 20.000 may be a bit too low in my opinion. Every folder in the mailbox is represented with three folders in the maildir (cur, new and tmp) and most of the time new and tmp will be empty.

[girish]

@fbartels sounds good. Shall I increase it to say 40000 ?

edit: will make it 50k for a nice round number.

[fbartels]

yes, that sounds good to me. For the moment I have increased the limit on my server by simply adding another zero.

[jagan]

I am getting this error too, as well as this new error in another of my cloudron:
QUote:
Backup failed: Too many executable files. Run "find /home/yellowtent/appsdata/8679c3b3-59b6-4d7e-a84a-d11dab4d5a19 -type f -executable" to investigate.

---

How can I solve both the empty directories as well as this error please? Thank you!

[girish]

@jagan can you check why you have so many executable files? If you run find /home/yellowtent/appsdata/8679c3b3-59b6-4d7e-a84a-d11dab4d5a19 -type f -executable it will give you a list of executable files. Usually, app data should not contain so many executables.

[jagan]

@girish sorry for the dumb question, but where do I run the 'find' command? I tried running it in the terminal of the nextcloud/wordpress app, but I got the 'no such file or directory' error.

[avatar1024]

@jagan looks like it's directly on your server (connecting ssh) and not on Cloudron interface

[shrey]

@girish
Hi. I've come across the same error, BoxError: Too many empty directories. Run "find /mnt/immich/i -type d -empty" to investigate, when i was investigating why my Immich app stopped updating at all.
It seems, that the backup process keeps failing and hence the app hasn't updated for the past month.

---

Btw:

root@apps:~# find /mnt/immich/i -type d -empty | wc -l
56108

[girish]

@shrey do you know why immich has 56k empty directories? Can you inspect those directories quickly? Something seems very wrong to have so many empty directories.

[shrey]

@girish No idea. Nor could i find any issue in their repo regarding anything related to this.

---

Could you recommend any workaround for now?

[shrey]

@girish
Additionally, my instance backups all are failing as well, just because the system can't backup one of the apps, Immich!

Is this the (designed) expected behaviour?