UNSOLVED Frequent 500s because of permission denied /app/code/tmp/cache
It looks like OpenProject is trying to write to a cache on most requests and then failing since Cloudron packages have /app/code set as read only. Is it possible to simlink /app/code/tmp to /app/data/tmp so that OpenProject caching works without these types of errors or to disable caching entirely?
Looks like some new behavior. In which view of openproject do you encounter this, so we can see how to reproduce and include that in our app tests.
@nebulon Clicking around any page (ie. enter a project, see work packages...) frequently encounter the above posted specific cache miss error or the below generic 500 making navigating the app nearly impossible. Often refreshing the page a few times does let it finally load successfully. I've increased the RAM to 2GB and have not seen any other errors that seem to indicate that these are results of tunable performance bottlenecks vs potentially some change in how the app does caching that is incompatible with the existing Cloudron packaging. Does it work seemingly "normal" in your test instances? Maybe my Cloudron box has another problem that's contributing.
I think what's happening is that
/tmp/cacheis probably a symlink into /tmp. We have a tmpcleaner which periodically cleans out temporary files inside app containers. We should probably symlink it to /run instead of /tmp . I have to check the package and confirm.
I was wrong. That directory is already symlinked to
/run/openproject_tmp. So the problem is something else.
@girish And to clarify,
/run/*isn't read-only for the app?
@adrw Yes, /run/ is writable. If you get the app into same error state, please do a ls -l on the directory and let's try to see what is happening.
I'm getting the same errors as @adrw as well.
This is on a fresh install of Ubuntu 20.04 and OpenProject.
This is the log:
Dec 24 15:22:40 E, [2020-12-24T08:46:40.026946 #259] ERROR -- : [current_user=Anonymous] Permission denied @ dir_s_mkdir - /app/code/tmp/cache/DD6/B70: Permission denied @ dir_s_mkdir - /app/code/tmp/cache/DD6/B70
And when I checked the permissions:
drwxr-xr-x 21 www-data www-data 4096 Dec 24 08:45 DD5/ drwxr-xr-x 3 root root 4096 Dec 24 06:10 DD6/ drwxr-xr-x 21 www-data www-data 4096 Dec 24 08:48 DD7/
Is it the issue with the directory,
DD6, being not writable by
Edit: There were 3 directories owned by
rootamong 49 directories.
@lolliop that is a good clue. The app should chown those directories on startup, however maybe they get created by some root process during runtime. If you restart the app, does the issue with that specific folder get sorted out (temporarily at least)?
So right after I restarted, the cache directory got emptied out and began creating 6 new directories which are all owned by
But after a few minutes, more directories appeared, and one of them was owned by
root. Again, after a few more minutes, more directories were created and another
rootowned directory appeared. All these happened without any user interactions (e.g. Clicking a link or logging in to OpenProject).
Here's another error I've encountered, similar to the above. Any ideas @nebulon ?
@adrw A quick and crude solution I found was to change the permissions of the files and directories under
docker exec -it OPENPROJECT-CONTAINER_ID chown -R www-data:www-data /app/code/tmp/cache
docier psto get the container ID of your OpenProject.
However, the issue here is that new
rootowned directories will be arbitrarily created from time to time depending on your usage, so that aforementioned command needs to be executed accordingly. I'm thinking to put it in a cron job and make it run every 5 minutes.
Please note that I've had only a few hours of testing of this, and I don't know how it can impact the system in the long run.
We definitely need a proper and permanent solution.
Reference here, but it doesn't really apply to our issues.
robi last edited by
@lolliop sounds like their ruby app isn't dropping privileges correctly or running in the right user context.