Since update to v8.2.1 backups fail with "Too many empty directories"

[shrey]

@nebulon said in Since update to v8.2.1 backups fail with "Too many empty directories":

so are all those empty dirs within the upload folder in your case?

Except for :

/mnt/immich/i/typesense

all of the paths seems to be:

/mnt/immich/i/upload/upload/b4fe288d-66ef-42ac-b74a-14ec12220a71/../..

---

@nebulon said in Since update to v8.2.1 backups fail with "Too many empty directories":

Also did you happen to change the data layout at some point in the past?

I think i migrated the app storage from local disk to a volume, around 5 months ago.

[nebulon]

Ah then this looks like an upstream bug https://github.com/immich-app/immich/issues/9202 with the discussion at https://github.com/immich-app/immich/discussions/9204 so looks like their strategy is that the user has to manually purge those.

[shrey]

@nebulon

Seem to have finally resolved the issue, by deleting those empty directories, using find . -type d -empty -delete, for now.

Although, the app is steadily creating new empty directories already!

[SebGG]

i have the same problem, but it stops while backuping my nextcloud instance. Is this a general issue?

[SebGG]

ok, i found the directory. It was the preview directory in the nextcloud instance. After cleaning the directory the backup was succesfull.

[jagan]

I have this backup issue with two instances.
In one with Nextcloud, it is empty directories.
In a Wordpress one, it is executable files.
Hoping to tackle this two this weekend.

[SebGG]

ok. but is this a bug or normal behavior caused by the cloudron update? i would say that these empty folders must have been there for some time and the backup only ran into this error last week.

[girish]

@SebGG this is normal behavior starting from 8.2 . we put this check in place to prevent backup or app data corruption .

• Too many empty folders. This means that something suspect is going on with the app . When you create too many empty dirs, you run out of inodes over time and this leads to fs corruption. Since empty dirs don't show up in du listings, people wonder why fs is becoming fullish when files are not taking space

• Too many executables. This is a security issue if data files are executable. In most cases, this is because user used a tarball which had incorrect permissions inside it . Or the mount permissions or wrong.

So, the backup failure is a way for end users to check on both the things above.

[SebGG]

ok, great!

[jagan]

@girish Hi Girish, I have this 'too many empty directories' issue on one of my Nextcloud instance and I haven't a clue how to deal with it. My last backup was on 6 Jan and I am getting nervous about it. Cloudron means I don't have to worry about backups, and lately, I an getting sleepless about this.

Can you please add an override for this error? Not having backups is IMHO, a more serious issue for me than any empty directories.

Also, running the command shows me directories that clearly are not empty in anyway. I don't know what else to do - I can't delete these directories precisely because they are not empty.

Any other pointers would be very helpful!

[joseph]

@jagan can you check the output of command find /app/data -type d -empty | wc -l in Web Terminal of nextcloud . If that is not a high number, then can you try back up again? The code is simply running that command and concluding the number is too high.

[imc67]

@joseph said in Since update to v8.2.1 backups fail with "Too many empty directories":

@jagan can you check the output of command find /app/data -type d -empty | wc -l in Web Terminal of nextcloud . If that is not a high number, then can you try back up again? The code is simply running that command and concluding the number is too high.

just curious, I don't have a backup issue but executed the command in web terminal NC: 12471 is this high?

and Immich: 26423

[jdaviescoates]

@imc67 said in Since update to v8.2.1 backups fail with "Too many empty directories":

just curious, I don't have a backup issue but executed the command in web terminal NC: 12471 is this high?

You made me curious too!

I seem to have 31267 (but also not currently having any backup issues either)

[jdaviescoates]

@joseph said in Since update to v8.2.1 backups fail with "Too many empty directories":

If that is not a high number,

Define high?

@joseph said in Since update to v8.2.1 backups fail with "Too many empty directories":

The code is simply running that command and concluding the number is too high.

How does the code define it?

[girish]

The number are just some pragmatic values. Empty dirs - 50000 , executables - 20000 and symlinks - 20000 . If it exceeds these, it will error.