Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Support
  3. out of space error leading to missing certs

out of space error leading to missing certs

Scheduled Pinned Locked Moved Solved Support
nginxdisk space
22 Posts 7 Posters 1.8k Views 7 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • R Offline
    R Offline
    roofboard
    wrote on last edited by
    #13

    @girish said in out of space error leading to missing certs:

    e-generate the nginx config for the dashboard alone. Once you have access to the dashboard, you can go to Location section of each app and click save. This will regenerate nginx config of the app

    FIXED!!!

    It is difficult to tell if deleting the conf files from the folder /etc/nginx/application and then restarting unbound Instructions then using systemctl restart nginx and systemctl restart box

    I say that because unbound definitely was not working at at one point.
    And as I remember nginx did start momentarily.

    However the solution came when I deleted the corrupted zero byte private key from the folder /home/yellowtent/platformdata/nginx/cert/

    When that file was deleted I was able to log in without ssl using firefox. Once in under the domains and certs section of cloudron I was able to click on Renew All Certs. That fixed SSL, and I was able to go into each program and re assign the dns settings by clicking save.

    scookeS 1 Reply Last reply
    3
    • nebulonN nebulon marked this topic as a question on
    • nebulonN nebulon has marked this topic as solved on
    • R roofboard

      @girish said in out of space error leading to missing certs:

      e-generate the nginx config for the dashboard alone. Once you have access to the dashboard, you can go to Location section of each app and click save. This will regenerate nginx config of the app

      FIXED!!!

      It is difficult to tell if deleting the conf files from the folder /etc/nginx/application and then restarting unbound Instructions then using systemctl restart nginx and systemctl restart box

      I say that because unbound definitely was not working at at one point.
      And as I remember nginx did start momentarily.

      However the solution came when I deleted the corrupted zero byte private key from the folder /home/yellowtent/platformdata/nginx/cert/

      When that file was deleted I was able to log in without ssl using firefox. Once in under the domains and certs section of cloudron I was able to click on Renew All Certs. That fixed SSL, and I was able to go into each program and re assign the dns settings by clicking save.

      scookeS Offline
      scookeS Offline
      scooke
      wrote on last edited by
      #14

      @roofboard Phew! Good work persisting, and thanks for sharing the solution.

      A life lived in fear is a life half-lived

      1 Reply Last reply
      0
      • subvenS Offline
        subvenS Offline
        subven
        wrote on last edited by
        #15

        @girish there is no way to trigger certificate renewal over the (SSH) console?

        I had a bug (a couple months ago) I never reported where stopped apps did not get a new cert and nginx failed to launch because of outdated/non valid certs making Cloudron brake (no nginx --> no dashboard) on system reboot. Fixed it by just copying over current cert files from working (non stopped) apps. They where obviously non valid for those stopped apps but I was able to start nginx, start the stopped apps and renew their certs.

        So in short: Would be nice to have a way to trigger cert renewal over console command and/or extend the troubleshoot guide with cert related stuff.

        R girishG robiR 3 Replies Last reply
        4
        • subvenS subven

          @girish there is no way to trigger certificate renewal over the (SSH) console?

          I had a bug (a couple months ago) I never reported where stopped apps did not get a new cert and nginx failed to launch because of outdated/non valid certs making Cloudron brake (no nginx --> no dashboard) on system reboot. Fixed it by just copying over current cert files from working (non stopped) apps. They where obviously non valid for those stopped apps but I was able to start nginx, start the stopped apps and renew their certs.

          So in short: Would be nice to have a way to trigger cert renewal over console command and/or extend the troubleshoot guide with cert related stuff.

          R Offline
          R Offline
          roofboard
          wrote on last edited by roofboard
          #16

          @girish

          Also this whole issue was caused by running out of space - I took a look at some of the other posts on out of space crashes and can tell it is a difficult problem to solve.

          Supposedly there is a running out of space warning but i never got that warning.

          I was thinking that a good solution for the running out of space error would involve taking the remaining space cron which calculates remaining space every 'n' minutes and integrating it over 'x' hours to arrive at time to disk full.

          This could relatively accurately predict if an out of space crash is pending or imminent - and if so... do things like stop processes prevent backup (if backing up to local filesystem) etc.

          Essentially

          1. predict the crash with a pinch of calculus.
          2. send a warning to the administrator.
          3. follow a contingency to protect the sever.

          Because I could imagine many ways this could happen, and my example is ONLY one way. A program can crash Cloudron I could have been copying video files, It could have been NextCloud, a spam attack on a mailserver.

          girishG 1 Reply Last reply
          1
          • R roofboard

            @girish

            Also this whole issue was caused by running out of space - I took a look at some of the other posts on out of space crashes and can tell it is a difficult problem to solve.

            Supposedly there is a running out of space warning but i never got that warning.

            I was thinking that a good solution for the running out of space error would involve taking the remaining space cron which calculates remaining space every 'n' minutes and integrating it over 'x' hours to arrive at time to disk full.

            This could relatively accurately predict if an out of space crash is pending or imminent - and if so... do things like stop processes prevent backup (if backing up to local filesystem) etc.

            Essentially

            1. predict the crash with a pinch of calculus.
            2. send a warning to the administrator.
            3. follow a contingency to protect the sever.

            Because I could imagine many ways this could happen, and my example is ONLY one way. A program can crash Cloudron I could have been copying video files, It could have been NextCloud, a spam attack on a mailserver.

            girishG Offline
            girishG Offline
            girish
            Staff
            wrote on last edited by
            #17

            @roofboard yes, agreed. I don't like it the way it currently right now that filling up disk space brings everything down. Currently, we have a simple cron checker which will give alerts if it's nearing some amount of disk space but this fails in many cases because it runs only every 6 hours or so (it's not run too often to prevent disk churn).

            I think a good long term solution is to figure out how to limit disk usage of apps. I think another thread there is a idea that maybe all appdata can be stored in a XFS partition. We can then enforce quotas on apps.

            robiR mehdiM 2 Replies Last reply
            1
            • subvenS subven

              @girish there is no way to trigger certificate renewal over the (SSH) console?

              I had a bug (a couple months ago) I never reported where stopped apps did not get a new cert and nginx failed to launch because of outdated/non valid certs making Cloudron brake (no nginx --> no dashboard) on system reboot. Fixed it by just copying over current cert files from working (non stopped) apps. They where obviously non valid for those stopped apps but I was able to start nginx, start the stopped apps and renew their certs.

              So in short: Would be nice to have a way to trigger cert renewal over console command and/or extend the troubleshoot guide with cert related stuff.

              girishG Offline
              girishG Offline
              girish
              Staff
              wrote on last edited by
              #18

              @subven said in out of space error leading to missing certs:

              nginx failed to launch because of outdated/non valid certs making Cloudron brake (no nginx --> no dashboard) on system reboot.

              Yes, indeed, this is a bug. As @roofboard also found out, the code check is a cert file exists but not if it's corrupt. I will get this fixed, so at the very least, restarting the box code will get the dashboard back up.

              1 Reply Last reply
              2
              • girishG girish

                @roofboard yes, agreed. I don't like it the way it currently right now that filling up disk space brings everything down. Currently, we have a simple cron checker which will give alerts if it's nearing some amount of disk space but this fails in many cases because it runs only every 6 hours or so (it's not run too often to prevent disk churn).

                I think a good long term solution is to figure out how to limit disk usage of apps. I think another thread there is a idea that maybe all appdata can be stored in a XFS partition. We can then enforce quotas on apps.

                robiR Offline
                robiR Offline
                robi
                wrote on last edited by
                #19

                @girish said in out of space error leading to missing certs:

                only every 6 hours or so

                The predictive aspect of @roofboard's suggestion is also a good one by tracking a bit of the rate of change, perhaps speeding up in frequency as we approach higher thresholds (>80%+) and slowing down when out of the danger zone(<80%).

                Combining this with an email to the admin which is more likely to be seen than a UI notification would be great, until we add the external mobile notification integration via external messaging services.. which is in the pipeline.

                Conscious tech

                1 Reply Last reply
                1
                • girishG girish

                  @roofboard yes, agreed. I don't like it the way it currently right now that filling up disk space brings everything down. Currently, we have a simple cron checker which will give alerts if it's nearing some amount of disk space but this fails in many cases because it runs only every 6 hours or so (it's not run too often to prevent disk churn).

                  I think a good long term solution is to figure out how to limit disk usage of apps. I think another thread there is a idea that maybe all appdata can be stored in a XFS partition. We can then enforce quotas on apps.

                  mehdiM Offline
                  mehdiM Offline
                  mehdi
                  App Dev
                  wrote on last edited by
                  #20

                  @girish said in out of space error leading to missing certs:

                  @roofboard yes, agreed. I don't like it the way it currently right now that filling up disk space brings everything down. Currently, we have a simple cron checker which will give alerts if it's nearing some amount of disk space but this fails in many cases because it runs only every 6 hours or so (it's not run too often to prevent disk churn).

                  I think a good long term solution is to figure out how to limit disk usage of apps. I think another thread there is a idea that maybe all appdata can be stored in a XFS partition. We can then enforce quotas on apps.

                  A good shorter term solution would be to allow to configure the level below which the alert is sent. Depending on if you use your server for storing text files, or if you download video, your "low disk" tolerance will be wildly different.

                  R 1 Reply Last reply
                  2
                  • mehdiM mehdi

                    @girish said in out of space error leading to missing certs:

                    @roofboard yes, agreed. I don't like it the way it currently right now that filling up disk space brings everything down. Currently, we have a simple cron checker which will give alerts if it's nearing some amount of disk space but this fails in many cases because it runs only every 6 hours or so (it's not run too often to prevent disk churn).

                    I think a good long term solution is to figure out how to limit disk usage of apps. I think another thread there is a idea that maybe all appdata can be stored in a XFS partition. We can then enforce quotas on apps.

                    A good shorter term solution would be to allow to configure the level below which the alert is sent. Depending on if you use your server for storing text files, or if you download video, your "low disk" tolerance will be wildly different.

                    R Offline
                    R Offline
                    roofboard
                    wrote on last edited by
                    #21

                    @mehdi

                    Yah... Maybe it would be enough just to give the administrator a few controls.

                    1. the cron interval
                    2. the warning level
                    3. the kill level (kill all modules)
                    1 Reply Last reply
                    1
                    • BrutalBirdieB BrutalBirdie referenced this topic on
                    • ei8fdbE ei8fdb referenced this topic on
                    • subvenS subven

                      @girish there is no way to trigger certificate renewal over the (SSH) console?

                      I had a bug (a couple months ago) I never reported where stopped apps did not get a new cert and nginx failed to launch because of outdated/non valid certs making Cloudron brake (no nginx --> no dashboard) on system reboot. Fixed it by just copying over current cert files from working (non stopped) apps. They where obviously non valid for those stopped apps but I was able to start nginx, start the stopped apps and renew their certs.

                      So in short: Would be nice to have a way to trigger cert renewal over console command and/or extend the troubleshoot guide with cert related stuff.

                      robiR Offline
                      robiR Offline
                      robi
                      wrote on last edited by robi
                      #22

                      @subven said in out of space error leading to missing certs:

                      @girish there is no way to trigger certificate renewal over the (SSH) console?

                      I'd like an answer to this question.. as I just ran into the missing cert problem too.

                      Having deleted all the conf/cert files, and gotten nginx started, the UI is still not accessible after box restart. All apps are inaccessible too.

                      box restart seems to recreate the /etc/nginx/applications/my.domain.conf BUT doesn't check if the /home/yellowtent/platformdata/nginx/certs/my.domain.host.cert is there.

                      How are they regenerated from the CLI?

                      Conscious tech

                      1 Reply Last reply
                      1
                      • robiR robi referenced this topic on
                      Reply
                      • Reply as topic
                      Log in to reply
                      • Oldest to Newest
                      • Newest to Oldest
                      • Most Votes


                      • Login

                      • Don't have an account? Register

                      • Login or register to search.
                      • First post
                        Last post
                      0
                      • Categories
                      • Recent
                      • Tags
                      • Popular
                      • Bookmarks
                      • Search