SOLVED Frequent 500s because of permission denied /app/code/tmp/cache

[adrw]

It looks like OpenProject is trying to write to a cache on most requests and then failing since Cloudron packages have /app/code set as read only. Is it possible to simlink /app/code/tmp to /app/data/tmp so that OpenProject caching works without these types of errors or to disable caching entirely?

[nebulon]

Looks like some new behavior. In which view of openproject do you encounter this, so we can see how to reproduce and include that in our app tests.

[adrw]

@nebulon Clicking around any page (ie. enter a project, see work packages...) frequently encounter the above posted specific cache miss error or the below generic 500 making navigating the app nearly impossible. Often refreshing the page a few times does let it finally load successfully. I've increased the RAM to 2GB and have not seen any other errors that seem to indicate that these are results of tunable performance bottlenecks vs potentially some change in how the app does caching that is incompatible with the existing Cloudron packaging. Does it work seemingly "normal" in your test instances? Maybe my Cloudron box has another problem that's contributing.

[adrw]

@adrw I did try the previous fix for other apps that have worked of hitting save in the Location to regenerate proxy and Nginx configuration and that seems to have helped a lot. I'll report back if I see the 500s again.

Thanks @girish for fixing this for the upcoming release.

[girish]

I think what's happening is that /tmp/cache is probably a symlink into /tmp. We have a tmpcleaner which periodically cleans out temporary files inside app containers. We should probably symlink it to /run instead of /tmp . I have to check the package and confirm.

[girish]

I was wrong. That directory is already symlinked to /run/openproject_tmp. So the problem is something else.

[adrw]

@girish And to clarify, /run/* isn't read-only for the app?

[girish]

@adrw Yes, /run/ is writable. If you get the app into same error state, please do a ls -l on the directory and let's try to see what is happening.

[lolliop]

I'm getting the same errors as @adrw as well.
This is on a fresh install of Ubuntu 20.04 and OpenProject.

This is the log:

Dec 24 15:22:40 E, [2020-12-24T08:46:40.026946 #259] ERROR -- : [current_user=Anonymous] Permission denied @ dir_s_mkdir - /app/code/tmp/cache/DD6/B70: Permission denied @ dir_s_mkdir - /app/code/tmp/cache/DD6/B70

And when I checked the permissions:

drwxr-xr-x 21 www-data www-data 4096 Dec 24 08:45 DD5/
drwxr-xr-x  3 root     root     4096 Dec 24 06:10 DD6/
drwxr-xr-x 21 www-data www-data 4096 Dec 24 08:48 DD7/

Is it the issue with the directory, DD6, being not writable by www-data?

Edit: There were 3 directories owned by root among 49 directories.

[nebulon]

@lolliop that is a good clue. The app should chown those directories on startup, however maybe they get created by some root process during runtime. If you restart the app, does the issue with that specific folder get sorted out (temporarily at least)?

[lolliop]

@nebulon,

So right after I restarted, the cache directory got emptied out and began creating 6 new directories which are all owned by www-data.

But after a few minutes, more directories appeared, and one of them was owned by root. Again, after a few more minutes, more directories were created and another root owned directory appeared. All these happened without any user interactions (e.g. Clicking a link or logging in to OpenProject).

[adrw]

Here's another error I've encountered, similar to the above. Any ideas @nebulon ?

[lolliop]

@adrw A quick and crude solution I found was to change the permissions of the files and directories under /app/code/tmp/cache:

docker exec -it OPENPROJECT-CONTAINER_ID chown -R www-data:www-data /app/code/tmp/cache

docier ps to get the container ID of your OpenProject.

However, the issue here is that new root owned directories will be arbitrarily created from time to time depending on your usage, so that aforementioned command needs to be executed accordingly. I'm thinking to put it in a cron job and make it run every 5 minutes.

Please note that I've had only a few hours of testing of this, and I don't know how it can impact the system in the long run.

We definitely need a proper and permanent solution.

Reference here, but it doesn't really apply to our issues.

[robi]

@lolliop sounds like their ruby app isn't dropping privileges correctly or running in the right user context.

[nebulon]

I have published a new app package with an attempt to fix the issue. Essentially all processes as well as apache2 only run as www-data so they don't have to drop privileges on their own. Especially phusion passenger for ruby gets loaded as root through apache, this is now fixed. Also all rake tasks are now run as www-data.

Please let us know if this indeed fixes the issue or not. I was not able to reliably reproduce the issue.

[Benoit]

Hi @nebulon,
i sent you an email because the last update of the openproject app doesn't work well. It can't restart after the update.
Log file : "Jan 22 09:46:31 `/var/www` is not writable."
Thanks for your help

[nebulon]

@benoit ah only saw this now after answering your support mail. This is expected since bundler should not run as root and thus should not have access to /var/www. In fact that folder is not really being utilized. Bundler anyways falls back to /tmp.

I didn't get the restart issue though, can you explain that one?

[Benoit]

@nebulon the app is running but we have this on the screen "This app is currently not responding. Try refreshing the page."
This happened this night after app updated.

[nebulon]

Oh I see. Since I had access to your instance for fixing the issue anyways, I saw that the app was healthy and running, but the nginx reverse proxy config wasn't correctly updated.
This may actually be a bug in Cloudron. The reason likely is that I've changed the app's internal port from 80 to 8000 to not require apache to run as root.

To fix this quickly, I've just submitted the location form of the app configure, which triggers a recreation of the reverse proxy files.

At least for me the app is now reachable.

[Benoit]

@nebulon yes it works now ! Great ! That it means this problem can occur on a new app install ? And what about a restart or restore of an openproject app ? Thanks

[nebulon]

@benoit this is only an issue once during that one update.

[nebulon]

So the was indeed a bug in the platform code with the port change. This is now fixed and will be part of the next Cloudron release.

[Peter Newman]

Based on my diagnosis, this is caused by the "worker" scheduler/cron task. Every time that runs (based on the log), I see a new file in /app/code/tmp/cache owned by root. Sometimes this new file is in a new directory (as the cache splits the storage across multiple directories on demand based on the random file name). When this newly created root owned folder is used by the normal process (running as www-data), it fails the permission check and users see the error message.

I've been "fixing" this by manually running

/app/code/tmp/cache# find -user root -exec chown www-data:www-data {} \;

at the terminal.

[nebulon]

@peter-newman ah great observation, I will release a new package with a fix then. I guess I can revert the apache and thus port change patch then.

[Benoit]

hi @nebulon, i don't know if it is due to your update but we encounter the not responding screen on openproject on an up to date and restarted server. I sent you an email.

[nebulon]

@benoit this is likely the same issue as after the other update, just recreate the nginx configs via submitting the domain/location settings without changing them in the app configure screen. This will be fixed with Cloudron 6.1

[Benoit]

@nebulon thanks for your explanation. it works now

[BrutalBirdie]

@nebulon said in Frequent 500s because of permission denied /app/code/tmp/cache:

Oh I see. Since I had access to your instance for fixing the issue anyways, I saw that the app was healthy and running, but the nginx reverse proxy config wasn't correctly updated.
This may actually be a bug in Cloudron. The reason likely is that I've changed the app's internal port from 80 to 8000 to not require apache to run as root.

To fix this quickly, I've just submitted the location form of the app configure, which triggers a recreation of the reverse proxy files.

At least for me the app is now reachable.

This explains a lot.