Since update to v8.2.1 backups fail with "Too many empty directories"

[girish]

@jagan can you check why you have so many executable files? If you run find /home/yellowtent/appsdata/8679c3b3-59b6-4d7e-a84a-d11dab4d5a19 -type f -executable it will give you a list of executable files. Usually, app data should not contain so many executables.

[jagan]

@girish sorry for the dumb question, but where do I run the 'find' command? I tried running it in the terminal of the nextcloud/wordpress app, but I got the 'no such file or directory' error.

[avatar1024]

@jagan looks like it's directly on your server (connecting ssh) and not on Cloudron interface

[shrey]

@girish
Hi. I've come across the same error, BoxError: Too many empty directories. Run "find /mnt/immich/i -type d -empty" to investigate, when i was investigating why my Immich app stopped updating at all.
It seems, that the backup process keeps failing and hence the app hasn't updated for the past month.

---

Btw:

root@apps:~# find /mnt/immich/i -type d -empty | wc -l
56108

[girish]

@shrey do you know why immich has 56k empty directories? Can you inspect those directories quickly? Something seems very wrong to have so many empty directories.

[shrey]

@girish No idea. Nor could i find any issue in their repo regarding anything related to this.

---

Could you recommend any workaround for now?

[shrey]

@girish
Additionally, my instance backups all are failing as well, just because the system can't backup one of the apps, Immich!

Is this the (designed) expected behaviour?

[girish]

@shrey you can disable immich backups and then the system backup will succeed. System backup === a backup that you can use to restore on another system . If one or more apps fail, it won't finish, this is just be design.

Did you find out what those empty folders are? Which paths inside immich are they in?

[jagan]

@girish I did run that, and got a very long list of files - php, png, svg on a wordpress site along with all the plugins.

Absolutely no idea how to proceed.

But for now, I will disable the apps shown the list and try, thank you!

[shrey]

@girish said in Since update to v8.2.1 backups fail with "Too many empty directories":

you can disable immich backups and then the system backup will succeed.

Yeah, that's the workaround i ended up with for now.

But i would really like to get my Immich start updating again (without compromising on the backup process). It's already 3 versions behind.

---

@girish said in Since update to v8.2.1 backups fail with "Too many empty directories":

Which paths inside immich are they in?

find /mnt/immich/i -type d -empty

/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/c7
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/96
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/01
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/09
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/36
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/2c
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/52
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/70
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/a3
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/93
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/33
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/0b
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/b7
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/9c
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/7d
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/3e
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/7c
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/d9
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/82
...

[girish]

@jagan said in Since update to v8.2.1 backups fail with "Too many empty directories":

Absolutely no idea how to proceed.

Those files are images (png). They should not be executable. You can go to the Web Terminal and run something like this:

find /app/data/public/wp-content/plugins/updraftplus -type f -exec chmod a-x {} +

Do this for other places which have 'x' bit on files as well. Generally, /app/data should not have any executable files (unless you have some scripts there).

[nebulon]

@shrey can you run the Migration job in the end of the jobs section within Immich? https://immich.app/docs/administration/jobs-workers#jobs

At least on my instance it purged most of the empty dirs. This seems to be a one-time housekeeping job. Not quite sure though why this is not run by default in Immich and needs to be manually triggered.

[shrey]

@nebulon said in Since update to v8.2.1 backups fail with "Too many empty directories":

@shrey can you run the Migration job in the end of the jobs section within Immich? https://immich.app/docs/administration/jobs-workers#jobs

Tried that.
But on my Immich instance (v1.123.0), it seems to have no effect on the concerned empty directories

root@apps:~# find /mnt/immich/i -type d -empty | wc -l
56587

[nebulon]

so are all those empty dirs within the upload folder in your case? Also did you happen to change the data layout at some point in the past?

[shrey]

@nebulon said in Since update to v8.2.1 backups fail with "Too many empty directories":

so are all those empty dirs within the upload folder in your case?

Except for :

/mnt/immich/i/typesense

all of the paths seems to be:

/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/../..

---

@nebulon said in Since update to v8.2.1 backups fail with "Too many empty directories":

Also did you happen to change the data layout at some point in the past?

I think i migrated the app storage from local disk to a volume, around 5 months ago.

[nebulon]

Ah then this looks like an upstream bug https://github.com/immich-app/immich/issues/9202 with the discussion at https://github.com/immich-app/immich/discussions/9204 so looks like their strategy is that the user has to manually purge those.

[shrey]

@nebulon

Seem to have finally resolved the issue, by deleting those empty directories, using find . -type d -empty -delete, for now.

Although, the app is steadily creating new empty directories already!

[SebGG]

i have the same problem, but it stops while backuping my nextcloud instance. Is this a general issue?

[SebGG]

ok, i found the directory. It was the preview directory in the nextcloud instance. After cleaning the directory the backup was succesfull.

[jagan]

I have this backup issue with two instances.
In one with Nextcloud, it is empty directories.
In a Wordpress one, it is executable files.
Hoping to tackle this two this weekend.