Help with improving GitLab startup latency

[timwedde]

I noticed that over time my GitLab instance startup times became slower and slower, but not always consistently. I eventually managed to track this down to a recursive directory ownership command that the startup script runs unconditionally:
chown -R git:git /app/data /run/gitlab /tmp/gitlab

My instance has about 350k files and directories in its artifact storage, which causes this command to take upwards of 3 minutes when filesystem caches are cold. Needless to say this is quite annoying when updating or restarting the app after some time, when the FS caches are cold. It only takes ~30s when the caches are hot, but this is rarely the case for updates for daily/weekly restarts. As such, I regularly lose 3-5 minutes on restarts and updates, which adds up over time.
This problem also gets worse as the instance grows, the more files and directories there are, the longer this command will take to run.

Is it required to run this command on every startup? If not, could it be made into a one-time thing or somehow be made conditional to only run when actually required? This would greatly reduce startup times (and thus downtime during updates) for me and presumably everyone else with larger instances.

[robi]

Ah yeah, from the chown it seems it is necessary for GitLab to start, but maybe not all at once.

If the command is run twice, once with depth 1 to allow GitLab to start, then later full depth if needed; that should solve the startup delay.

If there's any changes to the container/updates those permissions reset.