Frequent 500s because of permission denied /app/code/tmp/cache
-
So right after I restarted, the cache directory got emptied out and began creating 6 new directories which are all owned by
www-data.But after a few minutes, more directories appeared, and one of them was owned by
root. Again, after a few more minutes, more directories were created and anotherrootowned directory appeared. All these happened without any user interactions (e.g. Clicking a link or logging in to OpenProject). -
@adrw A quick and crude solution I found was to change the permissions of the files and directories under
/app/code/tmp/cache:docker exec -it OPENPROJECT-CONTAINER_ID chown -R www-data:www-data /app/code/tmp/cachedocier psto get the container ID of your OpenProject.However, the issue here is that new
rootowned directories will be arbitrarily created from time to time depending on your usage, so that aforementioned command needs to be executed accordingly. I'm thinking to put it in a cron job and make it run every 5 minutes.Please note that I've had only a few hours of testing of this, and I don't know how it can impact the system in the long run.
We definitely need a proper and permanent solution.
Reference here, but it doesn't really apply to our issues.
-
@adrw A quick and crude solution I found was to change the permissions of the files and directories under
/app/code/tmp/cache:docker exec -it OPENPROJECT-CONTAINER_ID chown -R www-data:www-data /app/code/tmp/cachedocier psto get the container ID of your OpenProject.However, the issue here is that new
rootowned directories will be arbitrarily created from time to time depending on your usage, so that aforementioned command needs to be executed accordingly. I'm thinking to put it in a cron job and make it run every 5 minutes.Please note that I've had only a few hours of testing of this, and I don't know how it can impact the system in the long run.
We definitely need a proper and permanent solution.
Reference here, but it doesn't really apply to our issues.
-
I have published a new app package with an attempt to fix the issue. Essentially all processes as well as apache2 only run as www-data so they don't have to drop privileges on their own. Especially phusion passenger for ruby gets loaded as root through apache, this is now fixed. Also all rake tasks are now run as www-data.
Please let us know if this indeed fixes the issue or not. I was not able to reliably reproduce the issue.
-
I have published a new app package with an attempt to fix the issue. Essentially all processes as well as apache2 only run as www-data so they don't have to drop privileges on their own. Especially phusion passenger for ruby gets loaded as root through apache, this is now fixed. Also all rake tasks are now run as www-data.
Please let us know if this indeed fixes the issue or not. I was not able to reliably reproduce the issue.
-
Hi @nebulon,
i sent you an email because the last update of the openproject app doesn't work well. It can't restart after the update.
Log file :"Jan 22 09:46:31 `/var/www` is not writable."
Thanks for your help@benoit ah only saw this now after answering your support mail. This is expected since bundler should not run as root and thus should not have access to
/var/www. In fact that folder is not really being utilized. Bundler anyways falls back to/tmp.I didn't get the restart issue though, can you explain that one?
-
@benoit ah only saw this now after answering your support mail. This is expected since bundler should not run as root and thus should not have access to
/var/www. In fact that folder is not really being utilized. Bundler anyways falls back to/tmp.I didn't get the restart issue though, can you explain that one?
-
Oh I see. Since I had access to your instance for fixing the issue anyways, I saw that the app was healthy and running, but the nginx reverse proxy config wasn't correctly updated.
This may actually be a bug in Cloudron. The reason likely is that I've changed the app's internal port from 80 to 8000 to not require apache to run as root.To fix this quickly, I've just submitted the location form of the app configure, which triggers a recreation of the reverse proxy files.
At least for me the app is now reachable.
-
Oh I see. Since I had access to your instance for fixing the issue anyways, I saw that the app was healthy and running, but the nginx reverse proxy config wasn't correctly updated.
This may actually be a bug in Cloudron. The reason likely is that I've changed the app's internal port from 80 to 8000 to not require apache to run as root.To fix this quickly, I've just submitted the location form of the app configure, which triggers a recreation of the reverse proxy files.
At least for me the app is now reachable.
-
@nebulon yes it works now ! Great ! That it means this problem can occur on a new app install ? And what about a restart or restore of an openproject app ? Thanks
-
Based on my diagnosis, this is caused by the "worker" scheduler/cron task. Every time that runs (based on the log), I see a new file in /app/code/tmp/cache owned by root. Sometimes this new file is in a new directory (as the cache splits the storage across multiple directories on demand based on the random file name). When this newly created root owned folder is used by the normal process (running as www-data), it fails the permission check and users see the error message.
I've been "fixing" this by manually running
/app/code/tmp/cache# find -user root -exec chown www-data:www-data {} \;at the terminal.
-
Based on my diagnosis, this is caused by the "worker" scheduler/cron task. Every time that runs (based on the log), I see a new file in /app/code/tmp/cache owned by root. Sometimes this new file is in a new directory (as the cache splits the storage across multiple directories on demand based on the random file name). When this newly created root owned folder is used by the normal process (running as www-data), it fails the permission check and users see the error message.
I've been "fixing" this by manually running
/app/code/tmp/cache# find -user root -exec chown www-data:www-data {} \;at the terminal.
-
@peter-newman ah great observation, I will release a new package with a fix then. I guess I can revert the apache and thus port change patch then.
-
hi @nebulon, i don't know if it is due to your update but we encounter the not responding screen on openproject on an up to date and restarted server. I sent you an email.
-
@benoit this is likely the same issue as after the other update, just recreate the nginx configs via submitting the domain/location settings without changing them in the app configure screen. This will be fixed with Cloudron 6.1
-
Oh I see. Since I had access to your instance for fixing the issue anyways, I saw that the app was healthy and running, but the nginx reverse proxy config wasn't correctly updated.
This may actually be a bug in Cloudron. The reason likely is that I've changed the app's internal port from 80 to 8000 to not require apache to run as root.To fix this quickly, I've just submitted the location form of the app configure, which triggers a recreation of the reverse proxy files.
At least for me the app is now reachable.
@nebulon said in Frequent 500s because of permission denied /app/code/tmp/cache:
Oh I see. Since I had access to your instance for fixing the issue anyways, I saw that the app was healthy and running, but the nginx reverse proxy config wasn't correctly updated.
This may actually be a bug in Cloudron. The reason likely is that I've changed the app's internal port from 80 to 8000 to not require apache to run as root.To fix this quickly, I've just submitted the location form of the app configure, which triggers a recreation of the reverse proxy files.
At least for me the app is now reachable.
This explains a lot.
