Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Support
  3. Graphite keeps crashing OOM

Graphite keeps crashing OOM

Scheduled Pinned Locked Moved Solved Support
graphsoom
37 Posts 6 Posters 5.1k Views 6 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • rmdesR Offline
      rmdesR Offline
      rmdes
      wrote on last edited by girish
      #1

      So every few hours Graphite crash, even when it has 2GB of memory, if I look at the log, this is what comes up : https://paste.armada.digital/todawelexi.sql

      it seems there is Python Twisted error that keeps repeating over and over again, my log file is 500+MB with a loop of the error in the link above.

      nebulonN 1 Reply Last reply
      0
      • rmdesR rmdes

        So every few hours Graphite crash, even when it has 2GB of memory, if I look at the log, this is what comes up : https://paste.armada.digital/todawelexi.sql

        it seems there is Python Twisted error that keeps repeating over and over again, my log file is 500+MB with a loop of the error in the link above.

        nebulonN Offline
        nebulonN Offline
        nebulon
        Staff
        wrote on last edited by
        #2

        @rmdes can you ensure that you run the latest Cloudron version 6.2.7 ? There were fixes for graphite in the last one.

        robiR rmdesR 2 Replies Last reply
        0
        • nebulonN nebulon

          @rmdes can you ensure that you run the latest Cloudron version 6.2.7 ? There were fixes for graphite in the last one.

          robiR Offline
          robiR Offline
          robi
          wrote on last edited by
          #3

          @nebulon I've had it happen on the latest v6.2.7 as well. Something keeps spiking its memory usage.

          Conscious tech

          1 Reply Last reply
          0
          • girishG Offline
            girishG Offline
            girish
            Staff
            wrote on last edited by
            #4

            Updating to 6.2.7 should definitely make the twisted errors sorted out. There was an error in graphite web configuration in previous releases.

            robiR 1 Reply Last reply
            0
            • girishG girish

              Updating to 6.2.7 should definitely make the twisted errors sorted out. There was an error in graphite web configuration in previous releases.

              robiR Offline
              robiR Offline
              robi
              wrote on last edited by
              #5

              @girish Happened twice today with 6.2.7.
              memory limit at 640MB down from 1250MB, where it had the same behavior.

              Conscious tech

              robiR 1 Reply Last reply
              0
              • robiR robi

                @girish Happened twice today with 6.2.7.
                memory limit at 640MB down from 1250MB, where it had the same behavior.

                robiR Offline
                robiR Offline
                robi
                wrote on last edited by
                #6

                @robi OOM twice more yesterday.

                It would be great if the message & email sent included the limit reached & time stamp.

                The email includes the email time though.

                Conscious tech

                1 Reply Last reply
                2
                • robiR Offline
                  robiR Offline
                  robi
                  wrote on last edited by
                  #7
                  Apr 07 09:15:39 builtins.StopIteration:
                  Apr 07 09:15:39 07/04/2021 16:15:39 :: [console] Unhandled Error
                  Apr 07 09:15:39 Traceback (most recent call last):
                  Apr 07 09:15:39 File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext
                  Apr 07 09:15:39 result = inContext.theWork()
                  Apr 07 09:15:39 File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
                  Apr 07 09:15:39 inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
                  Apr 07 09:15:39 File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
                  Apr 07 09:15:39 return self.currentContext().callWithContext(ctx, func, *args, **kw)
                  Apr 07 09:15:39 File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
                  Apr 07 09:15:39 return func(*args,**kw)
                  Apr 07 09:15:39 --- <exception caught here> ---
                  Apr 07 09:15:39 File "/usr/lib/python3/dist-packages/carbon/writer.py", line 189, in writeForever
                  Apr 07 09:15:39 writeCachedDataPoints()
                  Apr 07 09:15:39 File "/usr/lib/python3/dist-packages/carbon/writer.py", line 98, in writeCachedDataPoints
                  Apr 07 09:15:39 (metric, datapoints) = cache.drain_metric()
                  Apr 07 09:15:39 File "/usr/lib/python3/dist-packages/carbon/cache.py", line 187, in drain_metric
                  Apr 07 09:15:39 metric = self.strategy.choose_item()
                  Apr 07 09:15:39 File "/usr/lib/python3/dist-packages/carbon/cache.py", line 116, in choose_item
                  Apr 07 09:15:39 return next(self.queue)
                  Apr 07 09:15:39 builtins.StopIteration:
                  Apr 07 09:15:40 07/04/2021 16:15:40 :: [console] Unhandled Error
                  Apr 07 09:15:40 Traceback (most recent call last):
                  Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext
                  Apr 07 09:15:40 result = inContext.theWork()
                  Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
                  Apr 07 09:15:40 inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
                  Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
                  Apr 07 09:15:40 return self.currentContext().callWithContext(ctx, func, *args, **kw)
                  Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
                  Apr 07 09:15:40 return func(*args,**kw)
                  Apr 07 09:15:40 --- <exception caught here> ---
                  Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/carbon/writer.py", line 189, in writeForever
                  Apr 07 09:15:40 writeCachedDataPoints()
                  Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/carbon/writer.py", line 98, in writeCachedDataPoints
                  Apr 07 09:15:40 (metric, datapoints) = cache.drain_metric()
                  Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/carbon/cache.py", line 187, in drain_metric
                  Apr 07 09:15:40 metric = self.strategy.choose_item()
                  Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/carbon/cache.py", line 116, in choose_item
                  Apr 07 09:15:40 return next(self.queue)
                  Apr 07 09:15:40 builtins.StopIteration:
                  Apr 07 09:15:40 07/04/2021 16:15:40 :: [console] Unhandled Error
                  Apr 07 09:15:40 Traceback (most recent call last):
                  Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext
                  Apr 07 09:15:40 result = inContext.theWork()
                  Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
                  Apr 07 09:15:40 inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
                  Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
                  Apr 07 09:15:40 return self.currentContext().callWithContext(ctx, func, *args, **kw)
                  Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
                  Apr 07 09:15:40 return func(*args,**kw)
                  Apr 07 09:15:40 --- <exception caught here> ---
                  Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/carbon/writer.py", line 189, in writeForever
                  Apr 07 09:15:40 writeCachedDataPoints()
                  Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/carbon/writer.py", line 98, in writeCachedDataPoints
                  Apr 07 09:15:40 (metric, datapoints) = cache.drain_metric()
                  Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/carbon/cache.py", line 187, in drain_metric
                  Apr 07 09:15:40 metric = self.strategy.choose_item()
                  Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/carbon/cache.py", line 116, in choose_item
                  Apr 07 09:15:40 return next(self.queue)
                  Apr 07 09:15:40 builtins.StopIteration:
                  

                  It still runs out of memory with 2.5GB limit.

                  Also my browser tab for the logs starts timing out as if it can't keep up.. maybe it's in a continuous crash loop.

                  Conscious tech

                  rmdesR 1 Reply Last reply
                  1
                  • robiR robi
                    Apr 07 09:15:39 builtins.StopIteration:
                    Apr 07 09:15:39 07/04/2021 16:15:39 :: [console] Unhandled Error
                    Apr 07 09:15:39 Traceback (most recent call last):
                    Apr 07 09:15:39 File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext
                    Apr 07 09:15:39 result = inContext.theWork()
                    Apr 07 09:15:39 File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
                    Apr 07 09:15:39 inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
                    Apr 07 09:15:39 File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
                    Apr 07 09:15:39 return self.currentContext().callWithContext(ctx, func, *args, **kw)
                    Apr 07 09:15:39 File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
                    Apr 07 09:15:39 return func(*args,**kw)
                    Apr 07 09:15:39 --- <exception caught here> ---
                    Apr 07 09:15:39 File "/usr/lib/python3/dist-packages/carbon/writer.py", line 189, in writeForever
                    Apr 07 09:15:39 writeCachedDataPoints()
                    Apr 07 09:15:39 File "/usr/lib/python3/dist-packages/carbon/writer.py", line 98, in writeCachedDataPoints
                    Apr 07 09:15:39 (metric, datapoints) = cache.drain_metric()
                    Apr 07 09:15:39 File "/usr/lib/python3/dist-packages/carbon/cache.py", line 187, in drain_metric
                    Apr 07 09:15:39 metric = self.strategy.choose_item()
                    Apr 07 09:15:39 File "/usr/lib/python3/dist-packages/carbon/cache.py", line 116, in choose_item
                    Apr 07 09:15:39 return next(self.queue)
                    Apr 07 09:15:39 builtins.StopIteration:
                    Apr 07 09:15:40 07/04/2021 16:15:40 :: [console] Unhandled Error
                    Apr 07 09:15:40 Traceback (most recent call last):
                    Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext
                    Apr 07 09:15:40 result = inContext.theWork()
                    Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
                    Apr 07 09:15:40 inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
                    Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
                    Apr 07 09:15:40 return self.currentContext().callWithContext(ctx, func, *args, **kw)
                    Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
                    Apr 07 09:15:40 return func(*args,**kw)
                    Apr 07 09:15:40 --- <exception caught here> ---
                    Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/carbon/writer.py", line 189, in writeForever
                    Apr 07 09:15:40 writeCachedDataPoints()
                    Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/carbon/writer.py", line 98, in writeCachedDataPoints
                    Apr 07 09:15:40 (metric, datapoints) = cache.drain_metric()
                    Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/carbon/cache.py", line 187, in drain_metric
                    Apr 07 09:15:40 metric = self.strategy.choose_item()
                    Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/carbon/cache.py", line 116, in choose_item
                    Apr 07 09:15:40 return next(self.queue)
                    Apr 07 09:15:40 builtins.StopIteration:
                    Apr 07 09:15:40 07/04/2021 16:15:40 :: [console] Unhandled Error
                    Apr 07 09:15:40 Traceback (most recent call last):
                    Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext
                    Apr 07 09:15:40 result = inContext.theWork()
                    Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
                    Apr 07 09:15:40 inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
                    Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
                    Apr 07 09:15:40 return self.currentContext().callWithContext(ctx, func, *args, **kw)
                    Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
                    Apr 07 09:15:40 return func(*args,**kw)
                    Apr 07 09:15:40 --- <exception caught here> ---
                    Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/carbon/writer.py", line 189, in writeForever
                    Apr 07 09:15:40 writeCachedDataPoints()
                    Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/carbon/writer.py", line 98, in writeCachedDataPoints
                    Apr 07 09:15:40 (metric, datapoints) = cache.drain_metric()
                    Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/carbon/cache.py", line 187, in drain_metric
                    Apr 07 09:15:40 metric = self.strategy.choose_item()
                    Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/carbon/cache.py", line 116, in choose_item
                    Apr 07 09:15:40 return next(self.queue)
                    Apr 07 09:15:40 builtins.StopIteration:
                    

                    It still runs out of memory with 2.5GB limit.

                    Also my browser tab for the logs starts timing out as if it can't keep up.. maybe it's in a continuous crash loop.

                    rmdesR Offline
                    rmdesR Offline
                    rmdes
                    wrote on last edited by
                    #8

                    @robi same here, continued crash loop, log file is really huge

                    1 Reply Last reply
                    0
                    • nebulonN nebulon

                      @rmdes can you ensure that you run the latest Cloudron version 6.2.7 ? There were fixes for graphite in the last one.

                      rmdesR Offline
                      rmdesR Offline
                      rmdes
                      wrote on last edited by
                      #9

                      @nebulon running the latest cloudron version, confirmed

                      rmdesR 1 Reply Last reply
                      0
                      • rmdesR rmdes

                        @nebulon running the latest cloudron version, confirmed

                        rmdesR Offline
                        rmdesR Offline
                        rmdes
                        wrote on last edited by
                        #10

                        I don't have any visible python twisted package error anymore, but it seems the two last OOM event output this error : https://paste.armada.digital/enijufixep.coffeescript

                        1 Reply Last reply
                        0
                        • jdaviescoatesJ Online
                          jdaviescoatesJ Online
                          jdaviescoates
                          wrote on last edited by
                          #11

                          I'm also on 6.2.7 but graphite has just started crashing with OOM in the last couple of days (it's never crashed for me before then).

                          Also agree with @robi that the notifications within Cloudron really ought to include the time it happened (which I can only tell by seeing what time the email notification arrived).

                          I use Cloudron with Gandi & Hetzner

                          1 Reply Last reply
                          1
                          • robiR Offline
                            robiR Offline
                            robi
                            wrote on last edited by
                            #12

                            A manual restart by pushing the button in services, seems to have calmed down the crashing for now.

                            I hope the log rotator does it's job, we don't need to have so many huge logs around.

                            Conscious tech

                            1 Reply Last reply
                            1
                            • jdaviescoatesJ Online
                              jdaviescoatesJ Online
                              jdaviescoates
                              wrote on last edited by jdaviescoates
                              #13

                              Just had another Graphite OOM crash.

                              Seems strange seeing it's mostly just me using my Cloudron atm and I'm not really doing anything on it.

                              What does Graphite actually do?

                              I use Cloudron with Gandi & Hetzner

                              nebulonN 1 Reply Last reply
                              0
                              • jdaviescoatesJ jdaviescoates

                                Just had another Graphite OOM crash.

                                Seems strange seeing it's mostly just me using my Cloudron atm and I'm not really doing anything on it.

                                What does Graphite actually do?

                                nebulonN Offline
                                nebulonN Offline
                                nebulon
                                Staff
                                wrote on last edited by
                                #14

                                @jdaviescoates that service (graphite+collectd) collects the data used in the graphs, like memory usage over time. Given that it causes issues from time to time and also we don't really utilize it well, we are thinking of maybe collecting the data on our own and ditch graphite.

                                jdaviescoatesJ 1 Reply Last reply
                                2
                                • nebulonN nebulon

                                  @jdaviescoates that service (graphite+collectd) collects the data used in the graphs, like memory usage over time. Given that it causes issues from time to time and also we don't really utilize it well, we are thinking of maybe collecting the data on our own and ditch graphite.

                                  jdaviescoatesJ Online
                                  jdaviescoatesJ Online
                                  jdaviescoates
                                  wrote on last edited by
                                  #15

                                  Thanks

                                  @nebulon said in Graphite keeps crashing OOM:

                                  collecting the data on our own

                                  What would that look like?

                                  I use Cloudron with Gandi & Hetzner

                                  nebulonN 1 Reply Last reply
                                  0
                                  • jdaviescoatesJ jdaviescoates

                                    Thanks

                                    @nebulon said in Graphite keeps crashing OOM:

                                    collecting the data on our own

                                    What would that look like?

                                    nebulonN Offline
                                    nebulonN Offline
                                    nebulon
                                    Staff
                                    wrote on last edited by
                                    #16

                                    @jdaviescoates we don't know yet 😉

                                    scookeS 1 Reply Last reply
                                    1
                                    • nebulonN nebulon

                                      @jdaviescoates we don't know yet 😉

                                      scookeS Offline
                                      scookeS Offline
                                      scooke
                                      wrote on last edited by
                                      #17

                                      @nebulon Caprover uses Netdata... would that be possible?

                                      A life lived in fear is a life half-lived

                                      1 Reply Last reply
                                      2
                                      • robiR Offline
                                        robiR Offline
                                        robi
                                        wrote on last edited by
                                        #18

                                        0bcb80f1-c3a8-4e0d-af61-6a02f89d7332-image.png
                                        After a server restart, graphite won't start. Reconfig doesn't help.

                                        Conscious tech

                                        robiR 1 Reply Last reply
                                        0
                                        • robiR robi

                                          0bcb80f1-c3a8-4e0d-af61-6a02f89d7332-image.png
                                          After a server restart, graphite won't start. Reconfig doesn't help.

                                          robiR Offline
                                          robiR Offline
                                          robi
                                          wrote on last edited by
                                          #19

                                          I decided to reboot the box for security upgrades (from notifications) and it came up without errors this time.

                                          Conscious tech

                                          1 Reply Last reply
                                          0
                                          • jdaviescoatesJ Online
                                            jdaviescoatesJ Online
                                            jdaviescoates
                                            wrote on last edited by
                                            #20

                                            Graphite OOM, again.

                                            I use Cloudron with Gandi & Hetzner

                                            nebulonN 1 Reply Last reply
                                            0
                                            Reply
                                            • Reply as topic
                                            Log in to reply
                                            • Oldest to Newest
                                            • Newest to Oldest
                                            • Most Votes


                                              • Login

                                              • Don't have an account? Register

                                              • Login or register to search.
                                              • First post
                                                Last post
                                              0
                                              • Categories
                                              • Recent
                                              • Tags
                                              • Popular
                                              • Bookmarks
                                              • Search