Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Feature Requests
  3. Disk space should never bring a whole server down

Disk space should never bring a whole server down

Scheduled Pinned Locked Moved Feature Requests
50 Posts 15 Posters 6.7k Views 15 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • girishG girish

    @robi We actually have a disk space alert, in fact, it's there right now in the dashboard.

    57ac2165-226d-4f31-aa1e-a22cf131df92-image.png

    But the above is not super useful because it's just checking space in a cronjob. This cronjob is quite conservative because we don't want to keep spinning the disk too much. I am not aware of a way to get a "signal" from the server when disk space limits are hit. If a server fills up too fast between cron runs, the whole thing is useless...

    robiR Offline
    robiR Offline
    robi
    wrote on last edited by
    #25

    @girish yes, but does it email you when approaching the threshold?

    threshold setting? (twice a day should be plenty)

    action setting checkboxes? (maybe a custom one too?)

    heck, even deleting an non critical app would be fine since it's restorable from backup.

    Conscious tech

    1 Reply Last reply
    0
    • marcusquinnM marcusquinn

      I've triggered some bigger app data moves to the mounted 1TB volume but it seems to have chewed through 3GB of the remaining free space on the main volume already and I'm back to "Cloudron is offline. Reconnecting". Probably just making hasty tiredness errors now.

      scookeS Offline
      scookeS Offline
      scooke
      wrote on last edited by
      #26

      @marcusquinn Hang in there @marcusquinn. Bonne courage.

      A life lived in fear is a life half-lived

      1 Reply Last reply
      2
      • humptydumptyH Offline
        humptydumptyH Offline
        humptydumpty
        wrote on last edited by
        #27

        WHM has disk space limitations. Is it possible to copy their method and have it implemented in CR?

        1 Reply Last reply
        0
        • marcusquinnM Offline
          marcusquinnM Offline
          marcusquinn
          wrote on last edited by
          #28

          Thanks for all the help - I managed to get some extra hands on deck this morning and we're moving lots of data to a mounted volume for much more headroom.

          I still think it's a little too vulnerable having this hazard able to bring a server down.

          Also, I couldn't see if there's a way to set Email storage to be a mounted volume too?

          Web Design https://www.evergreen.je
          Development https://brandlight.org
          Life https://marcusquinn.com

          d19dotcaD 1 Reply Last reply
          1
          • girishG girish

            @robi We actually have a disk space alert, in fact, it's there right now in the dashboard.

            57ac2165-226d-4f31-aa1e-a22cf131df92-image.png

            But the above is not super useful because it's just checking space in a cronjob. This cronjob is quite conservative because we don't want to keep spinning the disk too much. I am not aware of a way to get a "signal" from the server when disk space limits are hit. If a server fills up too fast between cron runs, the whole thing is useless...

            mehdiM Offline
            mehdiM Offline
            mehdi
            App Dev
            wrote on last edited by
            #29

            @girish Also, the current warning is IMO not very useful if the threshold is not configurable. Depending on how the server is used, a few GB may be enough for weeks, or for mere hours if there's media stuff on the server, or if a user uploads stuff on nextcloud or something.

            1 Reply Last reply
            2
            • marcusquinnM marcusquinn

              Thanks for all the help - I managed to get some extra hands on deck this morning and we're moving lots of data to a mounted volume for much more headroom.

              I still think it's a little too vulnerable having this hazard able to bring a server down.

              Also, I couldn't see if there's a way to set Email storage to be a mounted volume too?

              d19dotcaD Online
              d19dotcaD Online
              d19dotca
              wrote on last edited by d19dotca
              #30

              @marcusquinn said in Disk space should never bring a whole server down:

              Also, I couldn't see if there's a way to set Email storage to be a mounted volume too?

              Currently, emails are part of boxdata and you need to move the boxdata entirely. I’ve done this in my current server due to the amount of email stored for my clients. The steps for this are at https://docs.cloudron.io/storage/#default-data-directory for reference.

              I’m making an assumption by volume you meant an external disk vs the actual Volumes function that Cloudron has.

              There is a feature request I believe to keep emails separate but boxdata really don’t contain much data at all other than emails so it’s doable as-is for now. It’d just be nice to see the GUI handle moving the email data much like it does for apps.

              --
              Dustin Dauncey
              www.d19.ca

              marcusquinnM 1 Reply Last reply
              2
              • d19dotcaD d19dotca

                @marcusquinn said in Disk space should never bring a whole server down:

                Also, I couldn't see if there's a way to set Email storage to be a mounted volume too?

                Currently, emails are part of boxdata and you need to move the boxdata entirely. I’ve done this in my current server due to the amount of email stored for my clients. The steps for this are at https://docs.cloudron.io/storage/#default-data-directory for reference.

                I’m making an assumption by volume you meant an external disk vs the actual Volumes function that Cloudron has.

                There is a feature request I believe to keep emails separate but boxdata really don’t contain much data at all other than emails so it’s doable as-is for now. It’d just be nice to see the GUI handle moving the email data much like it does for apps.

                marcusquinnM Offline
                marcusquinnM Offline
                marcusquinn
                wrote on last edited by
                #31

                @d19dotca Thanks. I'm an app specialist and anything more than a few minutes digging in the dirt is my kinda hell. Just getting brain fog now as I've lost a bunch of important work and 2 days of progress on it now 😞

                Web Design https://www.evergreen.je
                Development https://brandlight.org
                Life https://marcusquinn.com

                1 Reply Last reply
                0
                • marcusquinnM Offline
                  marcusquinnM Offline
                  marcusquinn
                  wrote on last edited by
                  #32

                  Anyone know where /app/data actually is in the full file system structure?

                  I'm trying to navigate a snapshot clone to see if that has the missing config.php file that hasn't come back for EspoCRM but just not seeing anything obvious and searching docs hasn't found me the clue.

                  Web Design https://www.evergreen.je
                  Development https://brandlight.org
                  Life https://marcusquinn.com

                  1 Reply Last reply
                  0
                  • marcusquinnM Offline
                    marcusquinnM Offline
                    marcusquinn
                    wrote on last edited by
                    #33

                    The problem I have is that EspoCRM Administration writes changes back to /app/data/data/config.php - however, that file also contains all the database connection details, password hash, basically everything for that instance to work.

                    So when the disk was full, it seems to have somehow written a 0kb version of config.php.

                    And because of the rsync encryption failing to backup EspoCRM, the Cloudron backups aren't complete.

                    So that leaves provider backup snapshot restore and dig around.

                    Basically, whatever anyone does - never allow the disk to get full - the cascade of problems that can happen from that interruption is just one massive time hole.

                    Web Design https://www.evergreen.je
                    Development https://brandlight.org
                    Life https://marcusquinn.com

                    marcusquinnM 1 Reply Last reply
                    1
                    • marcusquinnM marcusquinn

                      The problem I have is that EspoCRM Administration writes changes back to /app/data/data/config.php - however, that file also contains all the database connection details, password hash, basically everything for that instance to work.

                      So when the disk was full, it seems to have somehow written a 0kb version of config.php.

                      And because of the rsync encryption failing to backup EspoCRM, the Cloudron backups aren't complete.

                      So that leaves provider backup snapshot restore and dig around.

                      Basically, whatever anyone does - never allow the disk to get full - the cascade of problems that can happen from that interruption is just one massive time hole.

                      marcusquinnM Offline
                      marcusquinnM Offline
                      marcusquinn
                      wrote on last edited by
                      #34

                      @marcusquinn Holy sh*t, with some dumb-luck trying everything I know, I seem to have fixed it.

                      Lesson learnt - never run out of disk space - sods law says it will be the apps you rely on the most that will get corrupted.

                      Now, given the many open ways to load up a Cloudron with data (email/FilePizza/PrivateBin) maybe there's a way to avoid this causing a total fail?

                      Web Design https://www.evergreen.je
                      Development https://brandlight.org
                      Life https://marcusquinn.com

                      jdaviescoatesJ 1 Reply Last reply
                      3
                      • marcusquinnM marcusquinn

                        @marcusquinn Holy sh*t, with some dumb-luck trying everything I know, I seem to have fixed it.

                        Lesson learnt - never run out of disk space - sods law says it will be the apps you rely on the most that will get corrupted.

                        Now, given the many open ways to load up a Cloudron with data (email/FilePizza/PrivateBin) maybe there's a way to avoid this causing a total fail?

                        jdaviescoatesJ Offline
                        jdaviescoatesJ Offline
                        jdaviescoates
                        wrote on last edited by
                        #35

                        @marcusquinn said in Disk space should never bring a whole server down:

                        Now, given the many open ways to load up a Cloudron with data (email/FilePizza/PrivateBin) maybe there's a way to avoid this causing a total fail?

                        I think FilePizza if fully P2P and so I'm not sure you could fill the server up with that (but you could with Jirafeau).

                        But yeah, I reckon configurable disk space notifications (e.g. email/notify me hourly/daily/whatever once I've only got x space left) but be a good first step to help this not to happen.

                        I use Cloudron with Gandi & Hetzner

                        1 Reply Last reply
                        0
                        • marcusquinnM Offline
                          marcusquinnM Offline
                          marcusquinn
                          wrote on last edited by
                          #36

                          Quick fix idea: maybe 70% full is a better nag threshold?

                          Web Design https://www.evergreen.je
                          Development https://brandlight.org
                          Life https://marcusquinn.com

                          1 Reply Last reply
                          1
                          • bestknownhostB Offline
                            bestknownhostB Offline
                            bestknownhost
                            wrote on last edited by
                            #37

                            Thanks for all the feedback here. We discovered cloudron a whiles back and have been testing it out on a number of server over the last couple of months. We wanted to get a good handle on how everything works before rolling anything out into production. Firstly it’s a excellent platform and fills a great need. But we did run into a little problem with one of our test servers running on a digital ocean droplet. About 2 weeks ago it went from using 20gb of space to nearly 80gb in the space of 4 hours. We received an alart from digital ocean however things were happening so fast that all we could initially do is upgrade the instance, this gave us half and hour and then we had to do it again, then we just attached a 100gb volume. Although just in testing there was a wordpress app we were fond of and so we transferred it off the cloudron and left a pixelfed app. Somewhere between shutting down the server to add the volume and moving the Wordpress app, the space usage stopped increasing. I know what your thinking Wordpress right? No we checked the install before hand and it was working fine on another server. We then removed the 100gb volume and resized the digital ocean server back to its original size and evething was back to normal. I figured that some server updates ran that morning and some out of control process started this and resizing the server up and down somehow got rid of the problem.

                            robiR nebulonN 2 Replies Last reply
                            0
                            • bestknownhostB bestknownhost

                              Thanks for all the feedback here. We discovered cloudron a whiles back and have been testing it out on a number of server over the last couple of months. We wanted to get a good handle on how everything works before rolling anything out into production. Firstly it’s a excellent platform and fills a great need. But we did run into a little problem with one of our test servers running on a digital ocean droplet. About 2 weeks ago it went from using 20gb of space to nearly 80gb in the space of 4 hours. We received an alart from digital ocean however things were happening so fast that all we could initially do is upgrade the instance, this gave us half and hour and then we had to do it again, then we just attached a 100gb volume. Although just in testing there was a wordpress app we were fond of and so we transferred it off the cloudron and left a pixelfed app. Somewhere between shutting down the server to add the volume and moving the Wordpress app, the space usage stopped increasing. I know what your thinking Wordpress right? No we checked the install before hand and it was working fine on another server. We then removed the 100gb volume and resized the digital ocean server back to its original size and evething was back to normal. I figured that some server updates ran that morning and some out of control process started this and resizing the server up and down somehow got rid of the problem.

                              robiR Offline
                              robiR Offline
                              robi
                              wrote on last edited by
                              #38

                              @bestknownhost Did you perhaps have AdGuard installed?

                              Conscious tech

                              bestknownhostB 1 Reply Last reply
                              0
                              • robiR robi

                                @bestknownhost Did you perhaps have AdGuard installed?

                                bestknownhostB Offline
                                bestknownhostB Offline
                                bestknownhost
                                wrote on last edited by
                                #39

                                @robi No we didn't.

                                robiR 1 Reply Last reply
                                0
                                • bestknownhostB bestknownhost

                                  @robi No we didn't.

                                  robiR Offline
                                  robiR Offline
                                  robi
                                  wrote on last edited by
                                  #40

                                  @bestknownhost did you figure out what was filling up the disk with du -sh /* and drilling down?

                                  Conscious tech

                                  1 Reply Last reply
                                  1
                                  • bestknownhostB bestknownhost

                                    Thanks for all the feedback here. We discovered cloudron a whiles back and have been testing it out on a number of server over the last couple of months. We wanted to get a good handle on how everything works before rolling anything out into production. Firstly it’s a excellent platform and fills a great need. But we did run into a little problem with one of our test servers running on a digital ocean droplet. About 2 weeks ago it went from using 20gb of space to nearly 80gb in the space of 4 hours. We received an alart from digital ocean however things were happening so fast that all we could initially do is upgrade the instance, this gave us half and hour and then we had to do it again, then we just attached a 100gb volume. Although just in testing there was a wordpress app we were fond of and so we transferred it off the cloudron and left a pixelfed app. Somewhere between shutting down the server to add the volume and moving the Wordpress app, the space usage stopped increasing. I know what your thinking Wordpress right? No we checked the install before hand and it was working fine on another server. We then removed the 100gb volume and resized the digital ocean server back to its original size and evething was back to normal. I figured that some server updates ran that morning and some out of control process started this and resizing the server up and down somehow got rid of the problem.

                                    nebulonN Offline
                                    nebulonN Offline
                                    nebulon
                                    Staff
                                    wrote on last edited by
                                    #41

                                    @bestknownhost for a start to clarify, are you using an external backup storage or just the local disk for now? Using the local disk may cause disk usage to go up quickly depending on how much data you've put into the server.

                                    If that is not the case, then you may have hit some issue we recently saw with mysql binlogs https://forum.cloudron.io/topic/4510/able-to-clean-up-binlog-files-in-var-lib-mysql-directory?_=1616402616926 ?

                                    And as @robi mentioned, do you have any idea so far what is using all that disk space?

                                    1 Reply Last reply
                                    3
                                    • marcusquinnM marcusquinn

                                      Ugh, nope, disk full again, whole server down. Makes me dislike weekends when this stuff happens.

                                      No clue what to do but the post title remains valid - maybe Apps need diskspace limits because I'm caught between hard server resets, brief times of access, and then lockout again.

                                      D Offline
                                      D Offline
                                      DigitEgal
                                      wrote on last edited by DigitEgal
                                      #42

                                      @marcusquinn I was running into a simular issue while testing some stuff, most likely because of the Nexcloud Plugin "External Sites":
                                      I am not sure right now, but i dont think that it recreates the files, but more likely it writes a looooooooot of logs down since cpu got pushed aswell

                                      ( THATS NOT A TUTORIAL! ITS ONLY FOR REPEATABILITY OF BUGS! )
                                      How to create the Issue repeatable:
                                      1: Create a Nextcloud and share a folder(structure) to a public link.
                                      2. Insert this link into any secondary website (wordpress etc) as a button that does NOT open a new tab.
                                      3. Add the Plugin "External Sites" to Nextcloud - go to config and add the secondary website.
                                      4. By using the embed Mode of external site implementation this issue is possible to get triggered by a user with access to the External Sites Buttons.
                                      4.1 *Actual i did not test it by using a non-admin user as "trigger" user

                                      How to finally trigger the filling of Disk space ?
                                      -> Now follow the link in Nextcloud to your secondary website.
                                      -> By clicking the button back into nextcloud the issues is triggered.

                                      ( THATS NOT A TUTORIAL! ITS ONLY FOR REPEATABILITY OF BUGS! )

                                      1 Reply Last reply
                                      2
                                      • girishG girish referenced this topic on
                                      • M Offline
                                        M Offline
                                        makemrproper
                                        wrote on last edited by makemrproper
                                        #43

                                        Here is my SOLUTION:

                                        It does not solve the root cause why you are running out of space, but with this methodology you will buy yourself time.

                                        Generate 3 files of 2 gigabytes each.
                                        This is one way of generating these files:
                                        fallocate -l 2G /storage-padding-buffer-2-gb-file1.img
                                        fallocate -l 2G /storage-padding-buffer-2-gb-file2.img
                                        fallocate -l 2G /storage-padding-buffer-2-gb-file3.img

                                        When your server is out of storage, you may delete one or all of these padding files, so that regain the space you need to rescue the server.

                                        I have had the same issue with cloudron, because over time, storage will run out.
                                        For now I chose not to update the storage of my VPS server because it will double my hosting cost for this node, from USD400 to USD800 per year. That's digital ocean pricing for you, but I digress.

                                        This is a systems engineering issue and isn't caused by Cloudron. However I would not have anything against an elegant solution from the team if it were possible :).

                                        I want to say I am working on a post to describe I work with a massive cleanup, and exactly which steps I took to regain loads of space. TLDR; use ncdu, analyze all containers and identify where apps are storing logs and rotating these, clear NPM package cache in each container. More to cone

                                        Cloudron fan
                                        Paying Customer

                                        1 Reply Last reply
                                        6
                                        • marcusquinnM marcusquinn referenced this topic on
                                        • marcusquinnM Offline
                                          marcusquinnM Offline
                                          marcusquinn
                                          wrote on last edited by
                                          #44

                                          Maybe the Cloudron app needs to generate its own partition to run from, where regular app storage can't saturate the OS or Cloudron partitions?

                                          Web Design https://www.evergreen.je
                                          Development https://brandlight.org
                                          Life https://marcusquinn.com

                                          girishG 1 Reply Last reply
                                          2
                                          Reply
                                          • Reply as topic
                                          Log in to reply
                                          • Oldest to Newest
                                          • Newest to Oldest
                                          • Most Votes


                                          • Login

                                          • Don't have an account? Register

                                          • Login or register to search.
                                          • First post
                                            Last post
                                          0
                                          • Categories
                                          • Recent
                                          • Tags
                                          • Popular
                                          • Bookmarks
                                          • Search