Graphite keeps crashing OOM
-
So every few hours Graphite crash, even when it has 2GB of memory, if I look at the log, this is what comes up : https://paste.armada.digital/todawelexi.sql
it seems there is Python Twisted error that keeps repeating over and over again, my log file is 500+MB with a loop of the error in the link above.
-
Updating to 6.2.7 should definitely make the twisted errors sorted out. There was an error in graphite web configuration in previous releases.
-
Apr 07 09:15:39 builtins.StopIteration: Apr 07 09:15:39 07/04/2021 16:15:39 :: [console] Unhandled Error Apr 07 09:15:39 Traceback (most recent call last): Apr 07 09:15:39 File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext Apr 07 09:15:39 result = inContext.theWork() Apr 07 09:15:39 File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda> Apr 07 09:15:39 inContext.theWork = lambda: context.call(ctx, func, *args, **kw) Apr 07 09:15:39 File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext Apr 07 09:15:39 return self.currentContext().callWithContext(ctx, func, *args, **kw) Apr 07 09:15:39 File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext Apr 07 09:15:39 return func(*args,**kw) Apr 07 09:15:39 --- <exception caught here> --- Apr 07 09:15:39 File "/usr/lib/python3/dist-packages/carbon/writer.py", line 189, in writeForever Apr 07 09:15:39 writeCachedDataPoints() Apr 07 09:15:39 File "/usr/lib/python3/dist-packages/carbon/writer.py", line 98, in writeCachedDataPoints Apr 07 09:15:39 (metric, datapoints) = cache.drain_metric() Apr 07 09:15:39 File "/usr/lib/python3/dist-packages/carbon/cache.py", line 187, in drain_metric Apr 07 09:15:39 metric = self.strategy.choose_item() Apr 07 09:15:39 File "/usr/lib/python3/dist-packages/carbon/cache.py", line 116, in choose_item Apr 07 09:15:39 return next(self.queue) Apr 07 09:15:39 builtins.StopIteration: Apr 07 09:15:40 07/04/2021 16:15:40 :: [console] Unhandled Error Apr 07 09:15:40 Traceback (most recent call last): Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext Apr 07 09:15:40 result = inContext.theWork() Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda> Apr 07 09:15:40 inContext.theWork = lambda: context.call(ctx, func, *args, **kw) Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext Apr 07 09:15:40 return self.currentContext().callWithContext(ctx, func, *args, **kw) Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext Apr 07 09:15:40 return func(*args,**kw) Apr 07 09:15:40 --- <exception caught here> --- Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/carbon/writer.py", line 189, in writeForever Apr 07 09:15:40 writeCachedDataPoints() Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/carbon/writer.py", line 98, in writeCachedDataPoints Apr 07 09:15:40 (metric, datapoints) = cache.drain_metric() Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/carbon/cache.py", line 187, in drain_metric Apr 07 09:15:40 metric = self.strategy.choose_item() Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/carbon/cache.py", line 116, in choose_item Apr 07 09:15:40 return next(self.queue) Apr 07 09:15:40 builtins.StopIteration: Apr 07 09:15:40 07/04/2021 16:15:40 :: [console] Unhandled Error Apr 07 09:15:40 Traceback (most recent call last): Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext Apr 07 09:15:40 result = inContext.theWork() Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda> Apr 07 09:15:40 inContext.theWork = lambda: context.call(ctx, func, *args, **kw) Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext Apr 07 09:15:40 return self.currentContext().callWithContext(ctx, func, *args, **kw) Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext Apr 07 09:15:40 return func(*args,**kw) Apr 07 09:15:40 --- <exception caught here> --- Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/carbon/writer.py", line 189, in writeForever Apr 07 09:15:40 writeCachedDataPoints() Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/carbon/writer.py", line 98, in writeCachedDataPoints Apr 07 09:15:40 (metric, datapoints) = cache.drain_metric() Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/carbon/cache.py", line 187, in drain_metric Apr 07 09:15:40 metric = self.strategy.choose_item() Apr 07 09:15:40 File "/usr/lib/python3/dist-packages/carbon/cache.py", line 116, in choose_item Apr 07 09:15:40 return next(self.queue) Apr 07 09:15:40 builtins.StopIteration:
It still runs out of memory with 2.5GB limit.
Also my browser tab for the logs starts timing out as if it can't keep up.. maybe it's in a continuous crash loop.
-
I'm also on 6.2.7 but graphite has just started crashing with OOM in the last couple of days (it's never crashed for me before then).
Also agree with @robi that the notifications within Cloudron really ought to include the time it happened (which I can only tell by seeing what time the email notification arrived).
-
Just had another Graphite OOM crash.
Seems strange seeing it's mostly just me using my Cloudron atm and I'm not really doing anything on it.
What does Graphite actually do?
-
@jdaviescoates that service (graphite+collectd) collects the data used in the graphs, like memory usage over time. Given that it causes issues from time to time and also we don't really utilize it well, we are thinking of maybe collecting the data on our own and ditch graphite.
-
Thanks
@nebulon said in Graphite keeps crashing OOM:
collecting the data on our own
What would that look like?
-
@jdaviescoates we don't know yet
-
Graphite OOM, again.