Since update to v8.2.1 backups fail with "Too many empty directories"

fbartels · Jan 3, 2025, 8:52 AM

Hi,

this morning I noticed that since the 31st (and the update to 8.2.1 from 8.1.0) my backups keep failing with the same error message:

Too many empty directories. Run "find /home/yellowtent/boxdata/mail -type d -empty" to investigate

In the log this can be found for such a task:

2025-01-03T07:15:34.868Z box:shell backupformat/rsync: find /home/yellowtent/boxdata/mail -type d -empty
2025-01-03T07:15:41.125Z box:shell backupformat/rsync: find /home/yellowtent/boxdata/mail -type d -empty errored BoxError: find exited with code null signal SIGKILL
    at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:72:23)
    at ChildProcess.emit (node:events:519:28)
    at maybeClose (node:internal/child_process:1105:16)
    at Socket.<anonymous> (node:internal/child_process:457:11)
    at Socket.emit (node:events:519:28)
    at Pipe.<anonymous> (node:net:339:12) {
  reason: 'Shell Error',
  details: {},
  stdout: '/home/yellowtent/boxdata/mail/solr\n' +
[..]
    '/home/yellowtent/boxdata'... 2599152 more characters,
  stdoutLineCount: 20073,
  stderr: '',
  stderrLineCount: 0,
  code: null,
  signal: 'SIGKILL'
}
2025-01-03T07:15:41.205Z box:backupupload upload completed. error: BoxError: Too many empty directories. Run "find /home/yellowtent/boxdata/mail -type d -empty" to investigate at saveFsMetadata (/home/yellowtent/box/src/backupformat/rsync.js:131:82) at async Object.upload (/home/yellowtent/box/src/backupformat/rsync.js:270:5) at async Object.upload (/home/yellowtent/box/src/backuptask.js:92:5) { reason: 'FileSystem Error', details: {} }
2025-01-03T07:15:41.373Z box:backuptask runBackupUpload: result - {"result":"Too many empty directories. Run \"find /home/yellowtent/boxdata/mail -type d -empty\" to investigate"}
2025-01-03T07:15:41.533Z box:shell backuptask: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/mail rsync {"localRoot":"/home/yellowtent/boxdata/mail","layout":[]} errored BoxError: backuptask exited with code 50 signal null
    at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19)
    at ChildProcess.emit (node:events:519:28)
    at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
  reason: 'Shell Error',
  details: {},
  code: 50,
  signal: null
}
2025-01-03T07:15:41.612Z box:taskworker Task took 415.137 seconds
2025-01-03T07:15:41.614Z box:tasks setCompleted - 19010: {"result":null,"error":{"stack":"BoxError: Too many empty directories. Run \"find /home/yellowtent/boxdata/mail -type d -empty\" to investigate\n    at runBackupUpload (/home/yellowtent/box/src/backuptask.js:172:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async uploadMailSnapshot (/home/yellowtent/box/src/backuptask.js:413:5)\n    at async backupMailWithTag (/home/yellowtent/box/src/backuptask.js:462:5)\n    at async fullBackup (/home/yellowtent/box/src/backuptask.js:523:26)","name":"BoxError","reason":"External Error","details":{},"message":"Too many empty directories. Run \"find /home/yellowtent/boxdata/mail -type d -empty\" to investigate"}}
2025-01-03T07:15:41.615Z box:tasks update 19010: {"percent":100,"result":null,"error":{"stack":"BoxError: Too many empty directories. Run \"find /home/yellowtent/boxdata/mail -type d -empty\" to investigate\n    at runBackupUpload (/home/yellowtent/box/src/backuptask.js:172:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async uploadMailSnapshot (/home/yellowtent/box/src/backuptask.js:413:5)\n    at async backupMailWithTag (/home/yellowtent/box/src/backuptask.js:462:5)\n    at async fullBackup (/home/yellowtent/box/src/backuptask.js:523:26)","name":"BoxError","reason":"External Error","details":{},"message":"Too many empty directories. Run \"find /home/yellowtent/boxdata/mail -type d -empty\" to investigate"}}
BoxError: Too many empty directories. Run "find /home/yellowtent/boxdata/mail -type d -empty" to investigate
    at runBackupUpload (/home/yellowtent/box/src/backuptask.js:172:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async uploadMailSnapshot (/home/yellowtent/box/src/backuptask.js:413:5)
    at async backupMailWithTag (/home/yellowtent/box/src/backuptask.js:462:5)
    at async fullBackup (/home/yellowtent/box/src/backuptask.js:523:26)

What I am wondering about is the find exited with code null signal SIGKILL part. null seems a weird exit code and also running the find command manually finishes with exit code 0. As far as I can see the server should not be running out of resources during this and no related error seems to be in dmesg.

girish · wrote on Jan 3, 2025, 8:52 AM

@fbartels this comes from https://git.cloudron.io/platform/box/-/blob/master/src/backupformat/rsync.js?ref_type=heads#L130 . The command it runs is find /home/yellowtent/boxdata/mail -type d -empty and somehow this exceeds 20000 lines (in stdout or stderr). Can you check why by running it manually?

fbartels · Jan 3, 2025, 11:58 AM

I have a testing mailbox called "many-folders", which on its own already exceeds this limit:

find /home/yellowtent/boxdata/mail -type d -empty | grep many-folders | wc -l
23811

The total line count when not grepping for my test mailbox is 31099.

Depending on the amount of users and if these users are folder hoarders 20.000 may be a bit too low in my opinion. Every folder in the mailbox is represented with three folders in the maildir (cur, new and tmp) and most of the time new and tmp will be empty.

girish · Jan 3, 2025, 11:59 AM

@fbartels sounds good. Shall I increase it to say 40000 ?

edit: will make it 50k for a nice round number.

fbartels · wrote on Jan 3, 2025, 12:00 PM

yes, that sounds good to me. For the moment I have increased the limit on my server by simply adding another zero.

jagan · Jan 8, 2025, 5:01 PM

I am getting this error too, as well as this new error in another of my cloudron:
QUote:
Backup failed: Too many executable files. Run "find /home/yellowtent/appsdata/8679c3b3-59b6-4d7e-a84a-d11dab4d5a19 -type f -executable" to investigate.

How can I solve both the empty directories as well as this error please? Thank you!

girish · Jan 8, 2025, 5:02 PM

@jagan can you check why you have so many executable files? If you run find /home/yellowtent/appsdata/8679c3b3-59b6-4d7e-a84a-d11dab4d5a19 -type f -executable it will give you a list of executable files. Usually, app data should not contain so many executables.

jagan · Jan 8, 2025, 5:03 PM

@girish sorry for the dumb question, but where do I run the 'find' command? I tried running it in the terminal of the nextcloud/wordpress app, but I got the 'no such file or directory' error.

avatar1024 · J jagan Jan 8, 2025, 5:02 PM

@jagan looks like it's directly on your server (connecting ssh) and not on Cloudron interface

shrey · Jan 11, 2025, 9:17 AM

@girish
Hi. I've come across the same error, BoxError: Too many empty directories. Run "find /mnt/immich/i -type d -empty" to investigate, when i was investigating why my Immich app stopped updating at all.
It seems, that the backup process keeps failing and hence the app hasn't updated for the past month.

Btw:

root@apps:~# find /mnt/immich/i -type d -empty | wc -l
56108

girish · Jan 11, 2025, 11:27 AM

@shrey do you know why immich has 56k empty directories? Can you inspect those directories quickly? Something seems very wrong to have so many empty directories.

shrey · Jan 11, 2025, 2:44 PM

@girish No idea. Nor could i find any issue in their repo regarding anything related to this.

Could you recommend any workaround for now?

shrey · wrote on Jan 11, 2025, 5:54 PM

@girish
Additionally, my instance backups all are failing as well, just because the system can't backup one of the apps, Immich!

Is this the (designed) expected behaviour?

girish · Jan 13, 2025, 5:55 AM

@shrey you can disable immich backups and then the system backup will succeed. System backup === a backup that you can use to restore on another system . If one or more apps fail, it won't finish, this is just be design.

Did you find out what those empty folders are? Which paths inside immich are they in?

jagan · Jan 13, 2025, 5:40 AM

@girish I did run that, and got a very long list of files - php, png, svg on a wordpress site along with all the plugins.

Absolutely no idea how to proceed.

But for now, I will disable the apps shown the list and try, thank you!

shrey · Jan 13, 2025, 5:56 AM

@girish said in Since update to v8.2.1 backups fail with "Too many empty directories":

you can disable immich backups and then the system backup will succeed.

Yeah, that's the workaround i ended up with for now.

But i would really like to get my Immich start updating again (without compromising on the backup process). It's already 3 versions behind.

@girish said in Since update to v8.2.1 backups fail with "Too many empty directories":

Which paths inside immich are they in?

find /mnt/immich/i -type d -empty

/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/c7
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/96
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/01
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/09
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/36
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/2c
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/52
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/70
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/a3
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/93
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/33
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/0b
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/b7
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/9c
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/7d
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/3e
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/7c
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/d9
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/82
...

girish · J jagan Jan 13, 2025, 5:37 AM

@jagan said in Since update to v8.2.1 backups fail with "Too many empty directories":

Absolutely no idea how to proceed.

Those files are images (png). They should not be executable. You can go to the Web Terminal and run something like this:

find /app/data/public/wp-content/plugins/updraftplus -type f -exec chmod a-x {} +

Do this for other places which have 'x' bit on files as well. Generally, /app/data should not have any executable files (unless you have some scripts there).

nebulon · Jan 13, 2025, 10:28 AM

@shrey can you run the Migration job in the end of the jobs section within Immich? https://immich.app/docs/administration/jobs-workers#jobs

At least on my instance it purged most of the empty dirs. This seems to be a one-time housekeeping job. Not quite sure though why this is not run by default in Immich and needs to be manually triggered.

shrey · Jan 13, 2025, 10:29 AM

@nebulon said in Since update to v8.2.1 backups fail with "Too many empty directories":

@shrey can you run the Migration job in the end of the jobs section within Immich? https://immich.app/docs/administration/jobs-workers#jobs

Tried that.
But on my Immich instance (v1.123.0), it seems to have no effect on the concerned empty directories

root@apps:~# find /mnt/immich/i -type d -empty | wc -l
56587

nebulon · Jan 13, 2025, 11:00 AM

so are all those empty dirs within the upload folder in your case? Also did you happen to change the data layout at some point in the past?

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

Since update to v8.2.1 backups fail with "Too many empty directories"