Since update to v8.2.1 backups fail with "Too many empty directories"

[shrey]

@girish
Hi. I've come across the same error, BoxError: Too many empty directories. Run "find /mnt/immich/i -type d -empty" to investigate, when i was investigating why my Immich app stopped updating at all.
It seems, that the backup process keeps failing and hence the app hasn't updated for the past month.

---

Btw:

root@apps:~# find /mnt/immich/i -type d -empty | wc -l
56108

[girish]

@shrey do you know why immich has 56k empty directories? Can you inspect those directories quickly? Something seems very wrong to have so many empty directories.

[shrey]

@girish No idea. Nor could i find any issue in their repo regarding anything related to this.

---

Could you recommend any workaround for now?

[shrey]

@girish
Additionally, my instance backups all are failing as well, just because the system can't backup one of the apps, Immich!

Is this the (designed) expected behaviour?

[girish]

@shrey you can disable immich backups and then the system backup will succeed. System backup === a backup that you can use to restore on another system . If one or more apps fail, it won't finish, this is just be design.

Did you find out what those empty folders are? Which paths inside immich are they in?

[jagan]

@girish I did run that, and got a very long list of files - php, png, svg on a wordpress site along with all the plugins.

Absolutely no idea how to proceed.

But for now, I will disable the apps shown the list and try, thank you!

[shrey]

@girish said in Since update to v8.2.1 backups fail with "Too many empty directories":

you can disable immich backups and then the system backup will succeed.

Yeah, that's the workaround i ended up with for now.

But i would really like to get my Immich start updating again (without compromising on the backup process). It's already 3 versions behind.

---

@girish said in Since update to v8.2.1 backups fail with "Too many empty directories":

Which paths inside immich are they in?

find /mnt/immich/i -type d -empty

/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/c7
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/96
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/01
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/09
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/36
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/2c
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/52
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/70
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/a3
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/93
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/33
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/0b
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/b7
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/9c
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/7d
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/3e
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/7c
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/d9
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/82
...

[girish]

@jagan said in Since update to v8.2.1 backups fail with "Too many empty directories":

Absolutely no idea how to proceed.

Those files are images (png). They should not be executable. You can go to the Web Terminal and run something like this:

find /app/data/public/wp-content/plugins/updraftplus -type f -exec chmod a-x {} +

Do this for other places which have 'x' bit on files as well. Generally, /app/data should not have any executable files (unless you have some scripts there).

[nebulon]

@shrey can you run the Migration job in the end of the jobs section within Immich? https://immich.app/docs/administration/jobs-workers#jobs

At least on my instance it purged most of the empty dirs. This seems to be a one-time housekeeping job. Not quite sure though why this is not run by default in Immich and needs to be manually triggered.

[shrey]

@nebulon said in Since update to v8.2.1 backups fail with "Too many empty directories":

@shrey can you run the Migration job in the end of the jobs section within Immich? https://immich.app/docs/administration/jobs-workers#jobs

Tried that.
But on my Immich instance (v1.123.0), it seems to have no effect on the concerned empty directories

root@apps:~# find /mnt/immich/i -type d -empty | wc -l
56587

[nebulon]

so are all those empty dirs within the upload folder in your case? Also did you happen to change the data layout at some point in the past?

[shrey]

@nebulon said in Since update to v8.2.1 backups fail with "Too many empty directories":

so are all those empty dirs within the upload folder in your case?

Except for :

/mnt/immich/i/typesense

all of the paths seems to be:

/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/../..

---

@nebulon said in Since update to v8.2.1 backups fail with "Too many empty directories":

Also did you happen to change the data layout at some point in the past?

I think i migrated the app storage from local disk to a volume, around 5 months ago.

[nebulon]

Ah then this looks like an upstream bug https://github.com/immich-app/immich/issues/9202 with the discussion at https://github.com/immich-app/immich/discussions/9204 so looks like their strategy is that the user has to manually purge those.

[shrey]

@nebulon

Seem to have finally resolved the issue, by deleting those empty directories, using find . -type d -empty -delete, for now.

Although, the app is steadily creating new empty directories already!

[SebGG]

i have the same problem, but it stops while backuping my nextcloud instance. Is this a general issue?

[SebGG]

ok, i found the directory. It was the preview directory in the nextcloud instance. After cleaning the directory the backup was succesfull.

[jagan]

I have this backup issue with two instances.
In one with Nextcloud, it is empty directories.
In a Wordpress one, it is executable files.
Hoping to tackle this two this weekend.

[SebGG]

ok. but is this a bug or normal behavior caused by the cloudron update? i would say that these empty folders must have been there for some time and the backup only ran into this error last week.

[girish]

@SebGG this is normal behavior starting from 8.2 . we put this check in place to prevent backup or app data corruption .

• Too many empty folders. This means that something suspect is going on with the app . When you create too many empty dirs, you run out of inodes over time and this leads to fs corruption. Since empty dirs don't show up in du listings, people wonder why fs is becoming fullish when files are not taking space

• Too many executables. This is a security issue if data files are executable. In most cases, this is because user used a tarball which had incorrect permissions inside it . Or the mount permissions or wrong.

So, the backup failure is a way for end users to check on both the things above.

[SebGG]

ok, great!