Since update to v8.2.1 backups fail with "Too many empty directories"

[fbartels]

I have a testing mailbox called "many-folders", which on its own already exceeds this limit:

find /home/yellowtent/boxdata/mail -type d -empty | grep many-folders | wc -l
23811

The total line count when not grepping for my test mailbox is 31099.

Depending on the amount of users and if these users are folder hoarders 20.000 may be a bit too low in my opinion. Every folder in the mailbox is represented with three folders in the maildir (cur, new and tmp) and most of the time new and tmp will be empty.

[girish]

@fbartels sounds good. Shall I increase it to say 40000 ?

edit: will make it 50k for a nice round number.

[fbartels]

yes, that sounds good to me. For the moment I have increased the limit on my server by simply adding another zero.

[jagan]

I am getting this error too, as well as this new error in another of my cloudron:
QUote:
Backup failed: Too many executable files. Run "find /home/yellowtent/appsdata/8679c3b3-59b6-4d7e-a84a-d11dab4d5a19 -type f -executable" to investigate.

---

How can I solve both the empty directories as well as this error please? Thank you!

[girish]

@jagan can you check why you have so many executable files? If you run find /home/yellowtent/appsdata/8679c3b3-59b6-4d7e-a84a-d11dab4d5a19 -type f -executable it will give you a list of executable files. Usually, app data should not contain so many executables.

[jagan]

@girish sorry for the dumb question, but where do I run the 'find' command? I tried running it in the terminal of the nextcloud/wordpress app, but I got the 'no such file or directory' error.

[avatar1024]

@jagan looks like it's directly on your server (connecting ssh) and not on Cloudron interface

[shrey]

@girish
Hi. I've come across the same error, BoxError: Too many empty directories. Run "find /mnt/immich/i -type d -empty" to investigate, when i was investigating why my Immich app stopped updating at all.
It seems, that the backup process keeps failing and hence the app hasn't updated for the past month.

---

Btw:

root@apps:~# find /mnt/immich/i -type d -empty | wc -l
56108

[girish]

@shrey do you know why immich has 56k empty directories? Can you inspect those directories quickly? Something seems very wrong to have so many empty directories.

[shrey]

@girish No idea. Nor could i find any issue in their repo regarding anything related to this.

---

Could you recommend any workaround for now?

[shrey]

@girish
Additionally, my instance backups all are failing as well, just because the system can't backup one of the apps, Immich!

Is this the (designed) expected behaviour?

[girish]

@shrey you can disable immich backups and then the system backup will succeed. System backup === a backup that you can use to restore on another system . If one or more apps fail, it won't finish, this is just be design.

Did you find out what those empty folders are? Which paths inside immich are they in?

[jagan]

@girish I did run that, and got a very long list of files - php, png, svg on a wordpress site along with all the plugins.

Absolutely no idea how to proceed.

But for now, I will disable the apps shown the list and try, thank you!

[shrey]

@girish said in Since update to v8.2.1 backups fail with "Too many empty directories":

you can disable immich backups and then the system backup will succeed.

Yeah, that's the workaround i ended up with for now.

But i would really like to get my Immich start updating again (without compromising on the backup process). It's already 3 versions behind.

---

@girish said in Since update to v8.2.1 backups fail with "Too many empty directories":

Which paths inside immich are they in?

find /mnt/immich/i -type d -empty

/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/c7
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/96
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/01
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/09
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/36
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/2c
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/52
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/70
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/a3
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/93
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/33
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/0b
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/b7
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/9c
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/7d
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/3e
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/7c
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/d9
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/82
...

[girish]

@jagan said in Since update to v8.2.1 backups fail with "Too many empty directories":

Absolutely no idea how to proceed.

Those files are images (png). They should not be executable. You can go to the Web Terminal and run something like this:

find /app/data/public/wp-content/plugins/updraftplus -type f -exec chmod a-x {} +

Do this for other places which have 'x' bit on files as well. Generally, /app/data should not have any executable files (unless you have some scripts there).

[nebulon]

@shrey can you run the Migration job in the end of the jobs section within Immich? https://immich.app/docs/administration/jobs-workers#jobs

At least on my instance it purged most of the empty dirs. This seems to be a one-time housekeeping job. Not quite sure though why this is not run by default in Immich and needs to be manually triggered.

[shrey]

@nebulon said in Since update to v8.2.1 backups fail with "Too many empty directories":

@shrey can you run the Migration job in the end of the jobs section within Immich? https://immich.app/docs/administration/jobs-workers#jobs

Tried that.
But on my Immich instance (v1.123.0), it seems to have no effect on the concerned empty directories

root@apps:~# find /mnt/immich/i -type d -empty | wc -l
56587

[nebulon]

so are all those empty dirs within the upload folder in your case? Also did you happen to change the data layout at some point in the past?

[shrey]

@nebulon said in Since update to v8.2.1 backups fail with "Too many empty directories":

so are all those empty dirs within the upload folder in your case?

Except for :

/mnt/immich/i/typesense

all of the paths seems to be:

/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/../..

---

@nebulon said in Since update to v8.2.1 backups fail with "Too many empty directories":

Also did you happen to change the data layout at some point in the past?

I think i migrated the app storage from local disk to a volume, around 5 months ago.

[nebulon]

Ah then this looks like an upstream bug https://github.com/immich-app/immich/issues/9202 with the discussion at https://github.com/immich-app/immich/discussions/9204 so looks like their strategy is that the user has to manually purge those.