Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Support
  3. Graphite keeps crashing OOM

Graphite keeps crashing OOM

Scheduled Pinned Locked Moved Solved Support
graphsoom
37 Posts 6 Posters 5.1k Views 6 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • jdaviescoatesJ jdaviescoates

      Graphite OOM, again.

      nebulonN Offline
      nebulonN Offline
      nebulon
      Staff
      wrote on last edited by
      #21

      @jdaviescoates how much memory as the limit is set in your case? Also does the server itself have enough free memory to allocate? The settings in Cloudron are only the upper limit, but it may still get killed with oom if there is none available system-wide

      jdaviescoatesJ rmdesR 2 Replies Last reply
      0
      • nebulonN nebulon

        @jdaviescoates how much memory as the limit is set in your case? Also does the server itself have enough free memory to allocate? The settings in Cloudron are only the upper limit, but it may still get killed with oom if there is none available system-wide

        jdaviescoatesJ Offline
        jdaviescoatesJ Offline
        jdaviescoates
        wrote on last edited by
        #22

        @nebulon it was at whatever the default is (256MB?) I've now upped it to 512MB to see if that stops it. Plenty of spare RAM on the machine.

        I use Cloudron with Gandi & Hetzner

        1 Reply Last reply
        0
        • nebulonN nebulon

          @jdaviescoates how much memory as the limit is set in your case? Also does the server itself have enough free memory to allocate? The settings in Cloudron are only the upper limit, but it may still get killed with oom if there is none available system-wide

          rmdesR Offline
          rmdesR Offline
          rmdes
          wrote on last edited by
          #23

          @nebulon my graphite service has 1.60GB available, still OOM several times a day..
          the machine where cloudron is running has 30GB available, on average 15 Gb is being used leaving half of the available memory free.

          rmdesR 1 Reply Last reply
          1
          • nebulonN Offline
            nebulonN Offline
            nebulon
            Staff
            wrote on last edited by
            #24

            All this does not sound right then. Do you see anything suspicious in the graphite logs as such? Like frequent restarts of something or so?

            rmdesR 1 Reply Last reply
            1
            • nebulonN nebulon

              All this does not sound right then. Do you see anything suspicious in the graphite logs as such? Like frequent restarts of something or so?

              rmdesR Offline
              rmdesR Offline
              rmdes
              wrote on last edited by
              #25

              @nebulon This is the only errors I find in the log, beside the restarts :
              https://paste.armada.digital/xanopucuqu.sql

              robiR 1 Reply Last reply
              0
              • rmdesR rmdes

                @nebulon This is the only errors I find in the log, beside the restarts :
                https://paste.armada.digital/xanopucuqu.sql

                robiR Offline
                robiR Offline
                robi
                wrote on last edited by
                #26

                I get daily crashes too, with same/similar log messages about cache and draining issues.

                Conscious tech

                1 Reply Last reply
                0
                • rmdesR rmdes

                  @nebulon my graphite service has 1.60GB available, still OOM several times a day..
                  the machine where cloudron is running has 30GB available, on average 15 Gb is being used leaving half of the available memory free.

                  rmdesR Offline
                  rmdesR Offline
                  rmdes
                  wrote on last edited by
                  #27

                  my.armada.digital_.png
                  When graphite crash...

                  robiR 1 Reply Last reply
                  0
                  • rmdesR rmdes

                    my.armada.digital_.png
                    When graphite crash...

                    robiR Offline
                    robiR Offline
                    robi
                    wrote on last edited by
                    #28

                    @rmdes It's like Graphite sees Nessie the Loch Ness monster and freaks out..

                    Thanks for the graphs, er laughs. 😆

                    Conscious tech

                    rmdesR 1 Reply Last reply
                    2
                    • robiR robi

                      @rmdes It's like Graphite sees Nessie the Loch Ness monster and freaks out..

                      Thanks for the graphs, er laughs. 😆

                      rmdesR Offline
                      rmdesR Offline
                      rmdes
                      wrote on last edited by rmdes
                      #29

                      @robi here's another one, zoomed at 24h my.armada.digital_ (1).png
                      Funny thing is I understand it crashes because of memory issues (resulting out of python errors?)
                      but why/how does Graphite reboot itself ? I mean why fail to reboot for hours and suddenly it back online? why ?

                      robiR 1 Reply Last reply
                      0
                      • rmdesR rmdes

                        @robi here's another one, zoomed at 24h my.armada.digital_ (1).png
                        Funny thing is I understand it crashes because of memory issues (resulting out of python errors?)
                        but why/how does Graphite reboot itself ? I mean why fail to reboot for hours and suddenly it back online? why ?

                        robiR Offline
                        robiR Offline
                        robi
                        wrote on last edited by
                        #30

                        @rmdes nice.. yep not how a health monitored app should behave.

                        looks like something got stuck for a while then finally failed to get kicked again.

                        Conscious tech

                        1 Reply Last reply
                        0
                        • rmdesR Offline
                          rmdesR Offline
                          rmdes
                          wrote on last edited by
                          #31

                          Maybe this python error can help ? https://paste.armada.digital/ovurasajof.sql

                          girishG 1 Reply Last reply
                          0
                          • rmdesR rmdes

                            Maybe this python error can help ? https://paste.armada.digital/ovurasajof.sql

                            girishG Offline
                            girishG Offline
                            girish
                            Staff
                            wrote on last edited by
                            #32

                            @rmdes are you able to write to me on support@ and give me ssh access, so I can debug this? Would be good understand what's happening here.

                            rmdesR 1 Reply Last reply
                            1
                            • girishG girish

                              @rmdes are you able to write to me on support@ and give me ssh access, so I can debug this? Would be good understand what's happening here.

                              rmdesR Offline
                              rmdesR Offline
                              rmdes
                              wrote on last edited by
                              #33

                              @girish Yes of course, doing this now, SSH has been enabled.

                              girishG 2 Replies Last reply
                              1
                              • rmdesR rmdes

                                @girish Yes of course, doing this now, SSH has been enabled.

                                girishG Offline
                                girishG Offline
                                girish
                                Staff
                                wrote on last edited by
                                #34

                                @rmdes thanks for the access. it seems your server somehow hits this carbon cache bug - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=923464

                                1 Reply Last reply
                                2
                                • rmdesR rmdes

                                  @girish Yes of course, doing this now, SSH has been enabled.

                                  girishG Offline
                                  girishG Offline
                                  girish
                                  Staff
                                  wrote on last edited by
                                  #35

                                  @rmdes I have applied the patch in the bug report and it seems to fix the problem. I have applied change to your server locally. Will be in next release.

                                  rmdesR 1 Reply Last reply
                                  2
                                  • girishG girish

                                    @rmdes I have applied the patch in the bug report and it seems to fix the problem. I have applied change to your server locally. Will be in next release.

                                    rmdesR Offline
                                    rmdesR Offline
                                    rmdes
                                    wrote on last edited by rmdes
                                    #36

                                    @girish So this only hit on me ?
                                    Anyway, Thanks a lot for applying the patch locally and fixing the issue !

                                    girishG 1 Reply Last reply
                                    2
                                    • rmdesR rmdes

                                      @girish So this only hit on me ?
                                      Anyway, Thanks a lot for applying the patch locally and fixing the issue !

                                      girishG Offline
                                      girishG Offline
                                      girish
                                      Staff
                                      wrote on last edited by
                                      #37

                                      @rmdes yes, I am not sure why. It doesn't happen in any of our demo servers or managed services. Quite strange. It could also be that maybe others have hit it but have not noticed it (since it only causes a CPU spike..) but clearly it's a bug since it's been fixed upstream.

                                      1 Reply Last reply
                                      3
                                      Reply
                                      • Reply as topic
                                      Log in to reply
                                      • Oldest to Newest
                                      • Newest to Oldest
                                      • Most Votes


                                        • Login

                                        • Don't have an account? Register

                                        • Login or register to search.
                                        • First post
                                          Last post
                                        0
                                        • Categories
                                        • Recent
                                        • Tags
                                        • Popular
                                        • Bookmarks
                                        • Search