Since update to v8.2.1 backups fail with "Too many empty directories"

[jagan]

@girish I did run that, and got a very long list of files - php, png, svg on a wordpress site along with all the plugins.

Absolutely no idea how to proceed.

But for now, I will disable the apps shown the list and try, thank you!

[shrey]

@girish said in Since update to v8.2.1 backups fail with "Too many empty directories":

you can disable immich backups and then the system backup will succeed.

Yeah, that's the workaround i ended up with for now.

But i would really like to get my Immich start updating again (without compromising on the backup process). It's already 3 versions behind.

---

@girish said in Since update to v8.2.1 backups fail with "Too many empty directories":

Which paths inside immich are they in?

find /mnt/immich/i -type d -empty

/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/c7
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/96
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/01
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/09
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/36
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/2c
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/52
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/70
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/a3
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/93
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/33
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/0b
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/b7
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/9c
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/7d
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/3e
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/7c
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/d9
/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/82/82
...

[girish]

@jagan said in Since update to v8.2.1 backups fail with "Too many empty directories":

Absolutely no idea how to proceed.

Those files are images (png). They should not be executable. You can go to the Web Terminal and run something like this:

find /app/data/public/wp-content/plugins/updraftplus -type f -exec chmod a-x {} +

Do this for other places which have 'x' bit on files as well. Generally, /app/data should not have any executable files (unless you have some scripts there).

[nebulon]

@shrey can you run the Migration job in the end of the jobs section within Immich? https://immich.app/docs/administration/jobs-workers#jobs

At least on my instance it purged most of the empty dirs. This seems to be a one-time housekeeping job. Not quite sure though why this is not run by default in Immich and needs to be manually triggered.

[shrey]

@nebulon said in Since update to v8.2.1 backups fail with "Too many empty directories":

@shrey can you run the Migration job in the end of the jobs section within Immich? https://immich.app/docs/administration/jobs-workers#jobs

Tried that.
But on my Immich instance (v1.123.0), it seems to have no effect on the concerned empty directories

root@apps:~# find /mnt/immich/i -type d -empty | wc -l
56587

[nebulon]

so are all those empty dirs within the upload folder in your case? Also did you happen to change the data layout at some point in the past?

[shrey]

@nebulon said in Since update to v8.2.1 backups fail with "Too many empty directories":

so are all those empty dirs within the upload folder in your case?

Except for :

/mnt/immich/i/typesense

all of the paths seems to be:

/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/../..

---

@nebulon said in Since update to v8.2.1 backups fail with "Too many empty directories":

Also did you happen to change the data layout at some point in the past?

I think i migrated the app storage from local disk to a volume, around 5 months ago.

[nebulon]

Ah then this looks like an upstream bug https://github.com/immich-app/immich/issues/9202 with the discussion at https://github.com/immich-app/immich/discussions/9204 so looks like their strategy is that the user has to manually purge those.

[shrey]

@nebulon

Seem to have finally resolved the issue, by deleting those empty directories, using find . -type d -empty -delete, for now.

Although, the app is steadily creating new empty directories already!

[SebGG]

i have the same problem, but it stops while backuping my nextcloud instance. Is this a general issue?

[SebGG]

ok, i found the directory. It was the preview directory in the nextcloud instance. After cleaning the directory the backup was succesfull.

[jagan]

I have this backup issue with two instances.
In one with Nextcloud, it is empty directories.
In a Wordpress one, it is executable files.
Hoping to tackle this two this weekend.

[SebGG]

ok. but is this a bug or normal behavior caused by the cloudron update? i would say that these empty folders must have been there for some time and the backup only ran into this error last week.

[girish]

@SebGG this is normal behavior starting from 8.2 . we put this check in place to prevent backup or app data corruption .

• Too many empty folders. This means that something suspect is going on with the app . When you create too many empty dirs, you run out of inodes over time and this leads to fs corruption. Since empty dirs don't show up in du listings, people wonder why fs is becoming fullish when files are not taking space

• Too many executables. This is a security issue if data files are executable. In most cases, this is because user used a tarball which had incorrect permissions inside it . Or the mount permissions or wrong.

So, the backup failure is a way for end users to check on both the things above.

[SebGG]

ok, great!

[jagan]

@girish Hi Girish, I have this 'too many empty directories' issue on one of my Nextcloud instance and I haven't a clue how to deal with it. My last backup was on 6 Jan and I am getting nervous about it. Cloudron means I don't have to worry about backups, and lately, I an getting sleepless about this.

Can you please add an override for this error? Not having backups is IMHO, a more serious issue for me than any empty directories.

Also, running the command shows me directories that clearly are not empty in anyway. I don't know what else to do - I can't delete these directories precisely because they are not empty.

Any other pointers would be very helpful!

[joseph]

@jagan can you check the output of command find /app/data -type d -empty | wc -l in Web Terminal of nextcloud . If that is not a high number, then can you try back up again? The code is simply running that command and concluding the number is too high.

[imc67]

@joseph said in Since update to v8.2.1 backups fail with "Too many empty directories":

@jagan can you check the output of command find /app/data -type d -empty | wc -l in Web Terminal of nextcloud . If that is not a high number, then can you try back up again? The code is simply running that command and concluding the number is too high.

just curious, I don't have a backup issue but executed the command in web terminal NC: 12471 is this high?

and Immich: 26423

[jdaviescoates]

@imc67 said in Since update to v8.2.1 backups fail with "Too many empty directories":

just curious, I don't have a backup issue but executed the command in web terminal NC: 12471 is this high?

You made me curious too!

I seem to have 31267 (but also not currently having any backup issues either)

[jdaviescoates]

@joseph said in Since update to v8.2.1 backups fail with "Too many empty directories":

If that is not a high number,

Define high?

@joseph said in Since update to v8.2.1 backups fail with "Too many empty directories":

The code is simply running that command and concluding the number is too high.

How does the code define it?