Since update to v8.2.1 backups fail with "Too many empty directories"

[shrey]

@nebulon said in Since update to v8.2.1 backups fail with "Too many empty directories":

so are all those empty dirs within the upload folder in your case?

Except for :

/mnt/immich/i/typesense

all of the paths seems to be:

/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/../..

---

@nebulon said in Since update to v8.2.1 backups fail with "Too many empty directories":

Also did you happen to change the data layout at some point in the past?

I think i migrated the app storage from local disk to a volume, around 5 months ago.

[nebulon]

Ah then this looks like an upstream bug https://github.com/immich-app/immich/issues/9202 with the discussion at https://github.com/immich-app/immich/discussions/9204 so looks like their strategy is that the user has to manually purge those.

[shrey]

@nebulon

Seem to have finally resolved the issue, by deleting those empty directories, using find . -type d -empty -delete, for now.

Although, the app is steadily creating new empty directories already!

[SebGG]

i have the same problem, but it stops while backuping my nextcloud instance. Is this a general issue?

[SebGG]

ok, i found the directory. It was the preview directory in the nextcloud instance. After cleaning the directory the backup was succesfull.

[jagan]

I have this backup issue with two instances.
In one with Nextcloud, it is empty directories.
In a Wordpress one, it is executable files.
Hoping to tackle this two this weekend.

[SebGG]

ok. but is this a bug or normal behavior caused by the cloudron update? i would say that these empty folders must have been there for some time and the backup only ran into this error last week.

[girish]

@SebGG this is normal behavior starting from 8.2 . we put this check in place to prevent backup or app data corruption .

• Too many empty folders. This means that something suspect is going on with the app . When you create too many empty dirs, you run out of inodes over time and this leads to fs corruption. Since empty dirs don't show up in du listings, people wonder why fs is becoming fullish when files are not taking space

• Too many executables. This is a security issue if data files are executable. In most cases, this is because user used a tarball which had incorrect permissions inside it . Or the mount permissions or wrong.

So, the backup failure is a way for end users to check on both the things above.

[SebGG]

ok, great!

[jagan]

@girish Hi Girish, I have this 'too many empty directories' issue on one of my Nextcloud instance and I haven't a clue how to deal with it. My last backup was on 6 Jan and I am getting nervous about it. Cloudron means I don't have to worry about backups, and lately, I an getting sleepless about this.

Can you please add an override for this error? Not having backups is IMHO, a more serious issue for me than any empty directories.

Also, running the command shows me directories that clearly are not empty in anyway. I don't know what else to do - I can't delete these directories precisely because they are not empty.

Any other pointers would be very helpful!

[joseph]

@jagan can you check the output of command find /app/data -type d -empty | wc -l in Web Terminal of nextcloud . If that is not a high number, then can you try back up again? The code is simply running that command and concluding the number is too high.

[imc67]

@joseph said in Since update to v8.2.1 backups fail with "Too many empty directories":

@jagan can you check the output of command find /app/data -type d -empty | wc -l in Web Terminal of nextcloud . If that is not a high number, then can you try back up again? The code is simply running that command and concluding the number is too high.

just curious, I don't have a backup issue but executed the command in web terminal NC: 12471 is this high?

and Immich: 26423

[jdaviescoates]

@imc67 said in Since update to v8.2.1 backups fail with "Too many empty directories":

just curious, I don't have a backup issue but executed the command in web terminal NC: 12471 is this high?

You made me curious too!

I seem to have 31267 (but also not currently having any backup issues either)

[jdaviescoates]

@joseph said in Since update to v8.2.1 backups fail with "Too many empty directories":

If that is not a high number,

Define high?

@joseph said in Since update to v8.2.1 backups fail with "Too many empty directories":

The code is simply running that command and concluding the number is too high.

How does the code define it?

[girish]

The number are just some pragmatic values. Empty dirs - 50000 , executables - 20000 and symlinks - 20000 . If it exceeds these, it will error.

[mononym]

@joseph I hit 50968 empty directories Is there a way to solve this as it hinders proper backup processes ? Thanks

P.S.: I think the issue is discussed here: https://github.com/nextcloud/server/issues/46425

Edit: I deleted the content of the preview folder with rm -r app/data/appdata_*/preview/* and then ran sudo -u www-data php -f /app/code/occ files:scan-app-data as suggested in the link above. Ended up with 129 empty directories.

[joseph]

@mononym I recently had to clean up a customer's server . The preview dir had over 230k empty dirs!

# wc -l /tmp/deleted_dirs.txt 
231605 /tmp/deleted_dirs.txt

[jagan]

@joseph Yep, this was my installation

Thank you very much @joseph can't thank you enough.

[robi]

@joseph was there something added to the code to prevent the buildup of empty dirs?

[joseph]

@robi those directories were legit directories under user data and not created by some plugin . I think only @jagan can answer how so many directories can be created. Maybe they are synced from some external source.