Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Navigation

    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    SOLVED Frequent 500s because of permission denied /app/code/tmp/cache

    OpenProject
    8
    28
    199
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • nebulon
      nebulon Staff @lolliop last edited by

      @lolliop that is a good clue. The app should chown those directories on startup, however maybe they get created by some root process during runtime. If you restart the app, does the issue with that specific folder get sorted out (temporarily at least)?

      L 1 Reply Last reply Reply Quote 0
      • L
        lolliop @nebulon last edited by

        @nebulon,

        So right after I restarted, the cache directory got emptied out and began creating 6 new directories which are all owned by www-data.

        But after a few minutes, more directories appeared, and one of them was owned by root. Again, after a few more minutes, more directories were created and another root owned directory appeared. All these happened without any user interactions (e.g. Clicking a link or logging in to OpenProject).

        A 1 Reply Last reply Reply Quote 1
        • A
          adrw @lolliop last edited by

          Here's another error I've encountered, similar to the above. Any ideas @nebulon ?

          Screen Shot 2021-01-01 at 23.43.37.png

          L 1 Reply Last reply Reply Quote 0
          • L
            lolliop @adrw last edited by lolliop

            @adrw A quick and crude solution I found was to change the permissions of the files and directories under /app/code/tmp/cache:

            docker exec -it OPENPROJECT-CONTAINER_ID chown -R www-data:www-data /app/code/tmp/cache

            docier ps to get the container ID of your OpenProject.

            However, the issue here is that new root owned directories will be arbitrarily created from time to time depending on your usage, so that aforementioned command needs to be executed accordingly. I'm thinking to put it in a cron job and make it run every 5 minutes.

            Please note that I've had only a few hours of testing of this, and I don't know how it can impact the system in the long run.

            We definitely need a proper and permanent solution.

            Reference here, but it doesn't really apply to our issues.

            robi 1 Reply Last reply Reply Quote 1
            • robi
              robi @lolliop last edited by

              @lolliop sounds like their ruby app isn't dropping privileges correctly or running in the right user context.

              1 Reply Last reply Reply Quote 2
              • nebulon
                nebulon Staff last edited by

                I have published a new app package with an attempt to fix the issue. Essentially all processes as well as apache2 only run as www-data so they don't have to drop privileges on their own. Especially phusion passenger for ruby gets loaded as root through apache, this is now fixed. Also all rake tasks are now run as www-data.

                Please let us know if this indeed fixes the issue or not. I was not able to reliably reproduce the issue.

                Benoit 1 Reply Last reply Reply Quote 1
                • Benoit
                  Benoit @nebulon last edited by

                  Hi @nebulon,
                  i sent you an email because the last update of the openproject app doesn't work well. It can't restart after the update.
                  Log file : "Jan 22 09:46:31 `/var/www` is not writable."
                  Thanks for your help

                  nebulon 1 Reply Last reply Reply Quote 0
                  • nebulon
                    nebulon Staff @Benoit last edited by

                    @benoit ah only saw this now after answering your support mail. This is expected since bundler should not run as root and thus should not have access to /var/www. In fact that folder is not really being utilized. Bundler anyways falls back to /tmp.

                    I didn't get the restart issue though, can you explain that one?

                    Benoit 1 Reply Last reply Reply Quote 0
                    • Benoit
                      Benoit @nebulon last edited by

                      @nebulon the app is running but we have this on the screen "This app is currently not responding. Try refreshing the page."
                      This happened this night after app updated.

                      1 Reply Last reply Reply Quote 0
                      • nebulon
                        nebulon Staff last edited by

                        Oh I see. Since I had access to your instance for fixing the issue anyways, I saw that the app was healthy and running, but the nginx reverse proxy config wasn't correctly updated.
                        This may actually be a bug in Cloudron. The reason likely is that I've changed the app's internal port from 80 to 8000 to not require apache to run as root.

                        To fix this quickly, I've just submitted the location form of the app configure, which triggers a recreation of the reverse proxy files.

                        At least for me the app is now reachable.

                        Benoit BrutalBirdie 2 Replies Last reply Reply Quote 2
                        • Benoit
                          Benoit @nebulon last edited by

                          @nebulon yes it works now ! Great ! That it means this problem can occur on a new app install ? And what about a restart or restore of an openproject app ? Thanks

                          nebulon 1 Reply Last reply Reply Quote 0
                          • nebulon
                            nebulon Staff @Benoit last edited by

                            @benoit this is only an issue once during that one update.

                            1 Reply Last reply Reply Quote 1
                            • nebulon
                              nebulon Staff last edited by

                              So the was indeed a bug in the platform code with the port change. This is now fixed and will be part of the next Cloudron release.

                              1 Reply Last reply Reply Quote 1
                              • P
                                Peter Newman last edited by

                                Based on my diagnosis, this is caused by the "worker" scheduler/cron task. Every time that runs (based on the log), I see a new file in /app/code/tmp/cache owned by root. Sometimes this new file is in a new directory (as the cache splits the storage across multiple directories on demand based on the random file name). When this newly created root owned folder is used by the normal process (running as www-data), it fails the permission check and users see the error message.

                                I've been "fixing" this by manually running

                                /app/code/tmp/cache# find -user root -exec chown www-data:www-data {} \;
                                

                                at the terminal.

                                nebulon 1 Reply Last reply Reply Quote 0
                                • nebulon
                                  nebulon Staff @Peter Newman last edited by

                                  @peter-newman ah great observation, I will release a new package with a fix then. I guess I can revert the apache and thus port change patch then.

                                  Benoit 1 Reply Last reply Reply Quote 0
                                  • Benoit
                                    Benoit @nebulon last edited by

                                    hi @nebulon, i don't know if it is due to your update but we encounter the not responding screen on openproject on an up to date and restarted server. I sent you an email.

                                    nebulon 1 Reply Last reply Reply Quote 0
                                    • nebulon
                                      nebulon Staff @Benoit last edited by

                                      @benoit this is likely the same issue as after the other update, just recreate the nginx configs via submitting the domain/location settings without changing them in the app configure screen. This will be fixed with Cloudron 6.1

                                      Benoit 1 Reply Last reply Reply Quote 1
                                      • Benoit
                                        Benoit @nebulon last edited by

                                        @nebulon thanks for your explanation. it works now 🙂

                                        1 Reply Last reply Reply Quote 0
                                        • BrutalBirdie
                                          BrutalBirdie App Dev @nebulon last edited by

                                          @nebulon said in Frequent 500s because of permission denied /app/code/tmp/cache:

                                          Oh I see. Since I had access to your instance for fixing the issue anyways, I saw that the app was healthy and running, but the nginx reverse proxy config wasn't correctly updated.
                                          This may actually be a bug in Cloudron. The reason likely is that I've changed the app's internal port from 80 to 8000 to not require apache to run as root.

                                          To fix this quickly, I've just submitted the location form of the app configure, which triggers a recreation of the reverse proxy files.

                                          At least for me the app is now reachable.

                                          This explains a lot.

                                          1 Reply Last reply Reply Quote 0
                                          • First post
                                            Last post