Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Support
  3. Graphite keeps crashing OOM

Graphite keeps crashing OOM

Scheduled Pinned Locked Moved Solved Support
graphsoom
37 Posts 6 Posters 5.6k Views 6 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • rmdesR rmdes

    @nebulon running the latest cloudron version, confirmed

    rmdesR Offline
    rmdesR Offline
    rmdes
    wrote on last edited by
    #10

    I don't have any visible python twisted package error anymore, but it seems the two last OOM event output this error : https://paste.armada.digital/enijufixep.coffeescript

    1 Reply Last reply
    0
    • jdaviescoatesJ Offline
      jdaviescoatesJ Offline
      jdaviescoates
      wrote on last edited by
      #11

      I'm also on 6.2.7 but graphite has just started crashing with OOM in the last couple of days (it's never crashed for me before then).

      Also agree with @robi that the notifications within Cloudron really ought to include the time it happened (which I can only tell by seeing what time the email notification arrived).

      I use Cloudron with Gandi & Hetzner

      1 Reply Last reply
      1
      • robiR Offline
        robiR Offline
        robi
        wrote on last edited by
        #12

        A manual restart by pushing the button in services, seems to have calmed down the crashing for now.

        I hope the log rotator does it's job, we don't need to have so many huge logs around.

        Conscious tech

        1 Reply Last reply
        1
        • jdaviescoatesJ Offline
          jdaviescoatesJ Offline
          jdaviescoates
          wrote on last edited by jdaviescoates
          #13

          Just had another Graphite OOM crash.

          Seems strange seeing it's mostly just me using my Cloudron atm and I'm not really doing anything on it.

          What does Graphite actually do?

          I use Cloudron with Gandi & Hetzner

          nebulonN 1 Reply Last reply
          0
          • jdaviescoatesJ jdaviescoates

            Just had another Graphite OOM crash.

            Seems strange seeing it's mostly just me using my Cloudron atm and I'm not really doing anything on it.

            What does Graphite actually do?

            nebulonN Away
            nebulonN Away
            nebulon
            Staff
            wrote on last edited by
            #14

            @jdaviescoates that service (graphite+collectd) collects the data used in the graphs, like memory usage over time. Given that it causes issues from time to time and also we don't really utilize it well, we are thinking of maybe collecting the data on our own and ditch graphite.

            jdaviescoatesJ 1 Reply Last reply
            2
            • nebulonN nebulon

              @jdaviescoates that service (graphite+collectd) collects the data used in the graphs, like memory usage over time. Given that it causes issues from time to time and also we don't really utilize it well, we are thinking of maybe collecting the data on our own and ditch graphite.

              jdaviescoatesJ Offline
              jdaviescoatesJ Offline
              jdaviescoates
              wrote on last edited by
              #15

              Thanks

              @nebulon said in Graphite keeps crashing OOM:

              collecting the data on our own

              What would that look like?

              I use Cloudron with Gandi & Hetzner

              nebulonN 1 Reply Last reply
              0
              • jdaviescoatesJ jdaviescoates

                Thanks

                @nebulon said in Graphite keeps crashing OOM:

                collecting the data on our own

                What would that look like?

                nebulonN Away
                nebulonN Away
                nebulon
                Staff
                wrote on last edited by
                #16

                @jdaviescoates we don't know yet 😉

                scookeS 1 Reply Last reply
                1
                • nebulonN nebulon

                  @jdaviescoates we don't know yet 😉

                  scookeS Offline
                  scookeS Offline
                  scooke
                  wrote on last edited by
                  #17

                  @nebulon Caprover uses Netdata... would that be possible?

                  A life lived in fear is a life half-lived

                  1 Reply Last reply
                  2
                  • robiR Offline
                    robiR Offline
                    robi
                    wrote on last edited by
                    #18

                    0bcb80f1-c3a8-4e0d-af61-6a02f89d7332-image.png
                    After a server restart, graphite won't start. Reconfig doesn't help.

                    Conscious tech

                    robiR 1 Reply Last reply
                    0
                    • robiR robi

                      0bcb80f1-c3a8-4e0d-af61-6a02f89d7332-image.png
                      After a server restart, graphite won't start. Reconfig doesn't help.

                      robiR Offline
                      robiR Offline
                      robi
                      wrote on last edited by
                      #19

                      I decided to reboot the box for security upgrades (from notifications) and it came up without errors this time.

                      Conscious tech

                      1 Reply Last reply
                      0
                      • jdaviescoatesJ Offline
                        jdaviescoatesJ Offline
                        jdaviescoates
                        wrote on last edited by
                        #20

                        Graphite OOM, again.

                        I use Cloudron with Gandi & Hetzner

                        nebulonN 1 Reply Last reply
                        0
                        • jdaviescoatesJ jdaviescoates

                          Graphite OOM, again.

                          nebulonN Away
                          nebulonN Away
                          nebulon
                          Staff
                          wrote on last edited by
                          #21

                          @jdaviescoates how much memory as the limit is set in your case? Also does the server itself have enough free memory to allocate? The settings in Cloudron are only the upper limit, but it may still get killed with oom if there is none available system-wide

                          jdaviescoatesJ rmdesR 2 Replies Last reply
                          0
                          • nebulonN nebulon

                            @jdaviescoates how much memory as the limit is set in your case? Also does the server itself have enough free memory to allocate? The settings in Cloudron are only the upper limit, but it may still get killed with oom if there is none available system-wide

                            jdaviescoatesJ Offline
                            jdaviescoatesJ Offline
                            jdaviescoates
                            wrote on last edited by
                            #22

                            @nebulon it was at whatever the default is (256MB?) I've now upped it to 512MB to see if that stops it. Plenty of spare RAM on the machine.

                            I use Cloudron with Gandi & Hetzner

                            1 Reply Last reply
                            0
                            • nebulonN nebulon

                              @jdaviescoates how much memory as the limit is set in your case? Also does the server itself have enough free memory to allocate? The settings in Cloudron are only the upper limit, but it may still get killed with oom if there is none available system-wide

                              rmdesR Offline
                              rmdesR Offline
                              rmdes
                              wrote on last edited by
                              #23

                              @nebulon my graphite service has 1.60GB available, still OOM several times a day..
                              the machine where cloudron is running has 30GB available, on average 15 Gb is being used leaving half of the available memory free.

                              rmdesR 1 Reply Last reply
                              1
                              • nebulonN Away
                                nebulonN Away
                                nebulon
                                Staff
                                wrote on last edited by
                                #24

                                All this does not sound right then. Do you see anything suspicious in the graphite logs as such? Like frequent restarts of something or so?

                                rmdesR 1 Reply Last reply
                                1
                                • nebulonN nebulon

                                  All this does not sound right then. Do you see anything suspicious in the graphite logs as such? Like frequent restarts of something or so?

                                  rmdesR Offline
                                  rmdesR Offline
                                  rmdes
                                  wrote on last edited by
                                  #25

                                  @nebulon This is the only errors I find in the log, beside the restarts :
                                  https://paste.armada.digital/xanopucuqu.sql

                                  robiR 1 Reply Last reply
                                  0
                                  • rmdesR rmdes

                                    @nebulon This is the only errors I find in the log, beside the restarts :
                                    https://paste.armada.digital/xanopucuqu.sql

                                    robiR Offline
                                    robiR Offline
                                    robi
                                    wrote on last edited by
                                    #26

                                    I get daily crashes too, with same/similar log messages about cache and draining issues.

                                    Conscious tech

                                    1 Reply Last reply
                                    0
                                    • rmdesR rmdes

                                      @nebulon my graphite service has 1.60GB available, still OOM several times a day..
                                      the machine where cloudron is running has 30GB available, on average 15 Gb is being used leaving half of the available memory free.

                                      rmdesR Offline
                                      rmdesR Offline
                                      rmdes
                                      wrote on last edited by
                                      #27

                                      my.armada.digital_.png
                                      When graphite crash...

                                      robiR 1 Reply Last reply
                                      0
                                      • rmdesR rmdes

                                        my.armada.digital_.png
                                        When graphite crash...

                                        robiR Offline
                                        robiR Offline
                                        robi
                                        wrote on last edited by
                                        #28

                                        @rmdes It's like Graphite sees Nessie the Loch Ness monster and freaks out..

                                        Thanks for the graphs, er laughs. 😆

                                        Conscious tech

                                        rmdesR 1 Reply Last reply
                                        2
                                        • robiR robi

                                          @rmdes It's like Graphite sees Nessie the Loch Ness monster and freaks out..

                                          Thanks for the graphs, er laughs. 😆

                                          rmdesR Offline
                                          rmdesR Offline
                                          rmdes
                                          wrote on last edited by rmdes
                                          #29

                                          @robi here's another one, zoomed at 24h my.armada.digital_ (1).png
                                          Funny thing is I understand it crashes because of memory issues (resulting out of python errors?)
                                          but why/how does Graphite reboot itself ? I mean why fail to reboot for hours and suddenly it back online? why ?

                                          robiR 1 Reply Last reply
                                          0
                                          Reply
                                          • Reply as topic
                                          Log in to reply
                                          • Oldest to Newest
                                          • Newest to Oldest
                                          • Most Votes


                                          • Login

                                          • Don't have an account? Register

                                          • Login or register to search.
                                          • First post
                                            Last post
                                          0
                                          • Categories
                                          • Recent
                                          • Tags
                                          • Popular
                                          • Bookmarks
                                          • Search