Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Navigation

    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    SOLVED Frequent 500s because of permission denied /app/code/tmp/cache

    OpenProject
    8
    28
    197
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      adrw @adrw last edited by

      @adrw I did try the previous fix for other apps that have worked of hitting save in the Location to regenerate proxy and Nginx configuration and that seems to have helped a lot. I'll report back if I see the 500s again.

      Thanks @girish for fixing this for the upcoming release.

      1 Reply Last reply Reply Quote 0
      • girish
        girish Staff last edited by

        I think what's happening is that /tmp/cache is probably a symlink into /tmp. We have a tmpcleaner which periodically cleans out temporary files inside app containers. We should probably symlink it to /run instead of /tmp . I have to check the package and confirm.

        1 Reply Last reply Reply Quote 1
        • girish
          girish Staff last edited by

          I was wrong. That directory is already symlinked to /run/openproject_tmp. So the problem is something else.

          A 1 Reply Last reply Reply Quote 1
          • A
            adrw @girish last edited by

            @girish And to clarify, /run/* isn't read-only for the app?

            girish 1 Reply Last reply Reply Quote 0
            • girish
              girish Staff @adrw last edited by

              @adrw Yes, /run/ is writable. If you get the app into same error state, please do a ls -l on the directory and let's try to see what is happening.

              1 Reply Last reply Reply Quote 0
              • L
                lolliop last edited by lolliop

                I'm getting the same errors as @adrw as well.
                This is on a fresh install of Ubuntu 20.04 and OpenProject.

                This is the log:

                Dec 24 15:22:40 E, [2020-12-24T08:46:40.026946 #259] ERROR -- : [current_user=Anonymous] Permission denied @ dir_s_mkdir - /app/code/tmp/cache/DD6/B70: Permission denied @ dir_s_mkdir - /app/code/tmp/cache/DD6/B70

                And when I checked the permissions:

                drwxr-xr-x 21 www-data www-data 4096 Dec 24 08:45 DD5/
                drwxr-xr-x  3 root     root     4096 Dec 24 06:10 DD6/
                drwxr-xr-x 21 www-data www-data 4096 Dec 24 08:48 DD7/
                

                Is it the issue with the directory, DD6, being not writable by www-data?

                Edit: There were 3 directories owned by root among 49 directories.

                nebulon 1 Reply Last reply Reply Quote 0
                • nebulon
                  nebulon Staff @lolliop last edited by

                  @lolliop that is a good clue. The app should chown those directories on startup, however maybe they get created by some root process during runtime. If you restart the app, does the issue with that specific folder get sorted out (temporarily at least)?

                  L 1 Reply Last reply Reply Quote 0
                  • L
                    lolliop @nebulon last edited by

                    @nebulon,

                    So right after I restarted, the cache directory got emptied out and began creating 6 new directories which are all owned by www-data.

                    But after a few minutes, more directories appeared, and one of them was owned by root. Again, after a few more minutes, more directories were created and another root owned directory appeared. All these happened without any user interactions (e.g. Clicking a link or logging in to OpenProject).

                    A 1 Reply Last reply Reply Quote 1
                    • A
                      adrw @lolliop last edited by

                      Here's another error I've encountered, similar to the above. Any ideas @nebulon ?

                      Screen Shot 2021-01-01 at 23.43.37.png

                      L 1 Reply Last reply Reply Quote 0
                      • L
                        lolliop @adrw last edited by lolliop

                        @adrw A quick and crude solution I found was to change the permissions of the files and directories under /app/code/tmp/cache:

                        docker exec -it OPENPROJECT-CONTAINER_ID chown -R www-data:www-data /app/code/tmp/cache

                        docier ps to get the container ID of your OpenProject.

                        However, the issue here is that new root owned directories will be arbitrarily created from time to time depending on your usage, so that aforementioned command needs to be executed accordingly. I'm thinking to put it in a cron job and make it run every 5 minutes.

                        Please note that I've had only a few hours of testing of this, and I don't know how it can impact the system in the long run.

                        We definitely need a proper and permanent solution.

                        Reference here, but it doesn't really apply to our issues.

                        robi 1 Reply Last reply Reply Quote 1
                        • robi
                          robi @lolliop last edited by

                          @lolliop sounds like their ruby app isn't dropping privileges correctly or running in the right user context.

                          1 Reply Last reply Reply Quote 2
                          • nebulon
                            nebulon Staff last edited by

                            I have published a new app package with an attempt to fix the issue. Essentially all processes as well as apache2 only run as www-data so they don't have to drop privileges on their own. Especially phusion passenger for ruby gets loaded as root through apache, this is now fixed. Also all rake tasks are now run as www-data.

                            Please let us know if this indeed fixes the issue or not. I was not able to reliably reproduce the issue.

                            Benoit 1 Reply Last reply Reply Quote 1
                            • Benoit
                              Benoit @nebulon last edited by

                              Hi @nebulon,
                              i sent you an email because the last update of the openproject app doesn't work well. It can't restart after the update.
                              Log file : "Jan 22 09:46:31 `/var/www` is not writable."
                              Thanks for your help

                              nebulon 1 Reply Last reply Reply Quote 0
                              • nebulon
                                nebulon Staff @Benoit last edited by

                                @benoit ah only saw this now after answering your support mail. This is expected since bundler should not run as root and thus should not have access to /var/www. In fact that folder is not really being utilized. Bundler anyways falls back to /tmp.

                                I didn't get the restart issue though, can you explain that one?

                                Benoit 1 Reply Last reply Reply Quote 0
                                • Benoit
                                  Benoit @nebulon last edited by

                                  @nebulon the app is running but we have this on the screen "This app is currently not responding. Try refreshing the page."
                                  This happened this night after app updated.

                                  1 Reply Last reply Reply Quote 0
                                  • nebulon
                                    nebulon Staff last edited by

                                    Oh I see. Since I had access to your instance for fixing the issue anyways, I saw that the app was healthy and running, but the nginx reverse proxy config wasn't correctly updated.
                                    This may actually be a bug in Cloudron. The reason likely is that I've changed the app's internal port from 80 to 8000 to not require apache to run as root.

                                    To fix this quickly, I've just submitted the location form of the app configure, which triggers a recreation of the reverse proxy files.

                                    At least for me the app is now reachable.

                                    Benoit BrutalBirdie 2 Replies Last reply Reply Quote 2
                                    • Benoit
                                      Benoit @nebulon last edited by

                                      @nebulon yes it works now ! Great ! That it means this problem can occur on a new app install ? And what about a restart or restore of an openproject app ? Thanks

                                      nebulon 1 Reply Last reply Reply Quote 0
                                      • nebulon
                                        nebulon Staff @Benoit last edited by

                                        @benoit this is only an issue once during that one update.

                                        1 Reply Last reply Reply Quote 1
                                        • nebulon
                                          nebulon Staff last edited by

                                          So the was indeed a bug in the platform code with the port change. This is now fixed and will be part of the next Cloudron release.

                                          1 Reply Last reply Reply Quote 1
                                          • P
                                            Peter Newman last edited by

                                            Based on my diagnosis, this is caused by the "worker" scheduler/cron task. Every time that runs (based on the log), I see a new file in /app/code/tmp/cache owned by root. Sometimes this new file is in a new directory (as the cache splits the storage across multiple directories on demand based on the random file name). When this newly created root owned folder is used by the normal process (running as www-data), it fails the permission check and users see the error message.

                                            I've been "fixing" this by manually running

                                            /app/code/tmp/cache# find -user root -exec chown www-data:www-data {} \;
                                            

                                            at the terminal.

                                            nebulon 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post