Solved Slow startup caused by fixing permissions in start.sh

[vladimir.d]

Can we have an option to disable fixing file permissions on startup or fix permissions later?
It takes 2-3 hours to start Nextcloud app (3Tb, ~2.5 million files).

[girish]

@vladimir-d Thanks, I have pushed a fix so we change permissions only when needed. We made a similar fix in minio some time ago as well.

https://git.cloudron.io/cloudron/nextcloud-app/-/commit/33d4721e7e8fba1ac3322ef39a0a211df18cffc2

[girish]

@vladimir-d Thanks, I have pushed a fix so we change permissions only when needed. We made a similar fix in minio some time ago as well.

https://git.cloudron.io/cloudron/nextcloud-app/-/commit/33d4721e7e8fba1ac3322ef39a0a211df18cffc2

[vladimir.d]

Looks good, looking forward for a new Nextcloud release.
Thank you.

[vladimir.d]

@girish could you also review the code on lines 10-11 of start.sh.
and can you move start.sh on writable fs?

[girish]

@vladimir-d So far a start, I have made the chown a bit like find /app/data \( -type f -or -type d \) -and ! -user www-data -exec chown www-data:www-data '{}' \+. That essentially checks file ownership before chowing them. I have also added a flag to skip chown for restarts.

If these two changes are not enough, then we have to add some more sophisticated check if files are already of the correct ownership or not.

[girish]

@vladimir-d Can you check if things are better with the latest update?

[vladimir.d]

@girish it looks promising, but a few points I need to mention about:

1. There is Preview Generator app which creates a cache of thumbnails in /app/data/appdata_XXXXXXXXX/preview/. It also contains lots of small files - it took 30 minutes to process 300Gb files in the folder on SSD drive (chown or even du -hs). Better to exclude it from chowning as well.

2. A timeout occurs on time consuming file operations (i.e. chowning) so Cloudron shows 'Not responding' app status for a quite long time (3 hours in our case). It's a little misleading. So ideally to move all such these operations to another process in the background (supervisord or even cron) and make the app running whilst it's updating permissions.

3. A new app release is very welcomed even if it's a dirty hotfix with start.sh on writable FS (I'd comment 'chowning' out for now).

[vladimir.d]

@girish well, I've updated the app so it triggered chowning again.

I created the flag /run/nextcloud/chowned manually, killed the chown process and restarted the app. The upgrade downtime was just a few minutes this time.

Ideally to have a workaround for upgrades as well (a flag in /app/data?).

[girish]

@vladimir-d awesome, thanks for testing!

Ideally to have a workaround for upgrades as well (a flag in /app/data?).

Yes, agreed, I will make a change for the release after. You should be safe for the moment across restarts/reboots.

The flag cannot be in /app/data because it won't work for the 'restore' case. restore is done as root user, then start.sh fixes up perms. If flag is stored in /app/data, it would incorrectly skip the chown.

I think what can be done is to just sample the ownership of some well known files and chown accordingly.

[vladimir.d]

@girish It has never ended to backup Nextcloud app using the native cloudron backup routine in our case so we disabled them for this app at all. We use rclone to backup the data to S3 for this application. Restoring would be also done using rclone running under www-data user so there will be no issues with the ownership in our case.

Yes, agree the workaround could be updating ownership only of certain files and folders on start.
Maybe a system wide background service/job (maybe using AQMP) to fix permissions/ownership files when necessary in applications?

[girish]

I put a fix in https://git.cloudron.io/cloudron/nextcloud-app/-/commit/66292618931d5ecf4283f1c3fa7c8e64a68238f5

[vladimir.d]

@girish it looks good.
What would involve to move existent data to the new data directory (/app/data/data)?

[girish]

@vladimir-d there seems to be no easy way to migrate nextcloud to different data directory (atleast in an automated way), see https://help.nextcloud.com/t/howto-change-move-data-directory-after-installation/17170 . Looks like the put the absolute path in the "share" database fields.