<![CDATA[Graphite keeps crashing OOM]]>So every few hours Graphite crash, even when it has 2GB of memory, if I look at the log, this is what comes up : https://paste.armada.digital/todawelexi.sql

it seems there is Python Twisted error that keeps repeating over and over again, my log file is 500+MB with a loop of the error in the link above.

]]>https://forum.cloudron.io/topic/4797/graphite-keeps-crashing-oomRSS for NodeTue, 14 Apr 2026 09:38:29 GMTWed, 31 Mar 2021 11:14:07 GMT60<![CDATA[Reply to Graphite keeps crashing OOM on Fri, 30 Apr 2021 16:21:12 GMT]]>@rmdes yes, I am not sure why. It doesn't happen in any of our demo servers or managed services. Quite strange. It could also be that maybe others have hit it but have not noticed it (since it only causes a CPU spike..) but clearly it's a bug since it's been fixed upstream.

]]>https://forum.cloudron.io/post/30606https://forum.cloudron.io/post/30606Fri, 30 Apr 2021 16:21:12 GMT<![CDATA[Reply to Graphite keeps crashing OOM on Fri, 30 Apr 2021 12:58:55 GMT]]>@girish So this only hit on me ?
Anyway, Thanks a lot for applying the patch locally and fixing the issue !

]]>https://forum.cloudron.io/post/30586https://forum.cloudron.io/post/30586Fri, 30 Apr 2021 12:58:55 GMT<![CDATA[Reply to Graphite keeps crashing OOM on Tue, 27 Apr 2021 21:29:43 GMT]]>@rmdes I have applied the patch in the bug report and it seems to fix the problem. I have applied change to your server locally. Will be in next release.

]]>https://forum.cloudron.io/post/30459https://forum.cloudron.io/post/30459Tue, 27 Apr 2021 21:29:43 GMT<![CDATA[Reply to Graphite keeps crashing OOM on Tue, 27 Apr 2021 16:08:21 GMT]]>@rmdes thanks for the access. it seems your server somehow hits this carbon cache bug - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=923464

]]>https://forum.cloudron.io/post/30441https://forum.cloudron.io/post/30441Tue, 27 Apr 2021 16:08:21 GMT<![CDATA[Reply to Graphite keeps crashing OOM on Tue, 27 Apr 2021 07:56:47 GMT]]>@girish Yes of course, doing this now, SSH has been enabled.

]]>https://forum.cloudron.io/post/30428https://forum.cloudron.io/post/30428Tue, 27 Apr 2021 07:56:47 GMT<![CDATA[Reply to Graphite keeps crashing OOM on Mon, 26 Apr 2021 20:09:11 GMT]]>@rmdes are you able to write to me on support@ and give me ssh access, so I can debug this? Would be good understand what's happening here.

]]>https://forum.cloudron.io/post/30408https://forum.cloudron.io/post/30408Mon, 26 Apr 2021 20:09:11 GMT<![CDATA[Reply to Graphite keeps crashing OOM on Mon, 26 Apr 2021 15:18:31 GMT]]>Maybe this python error can help ? https://paste.armada.digital/ovurasajof.sql

]]>https://forum.cloudron.io/post/30388https://forum.cloudron.io/post/30388Mon, 26 Apr 2021 15:18:31 GMT<![CDATA[Reply to Graphite keeps crashing OOM on Sat, 24 Apr 2021 15:55:13 GMT]]>@rmdes nice.. yep not how a health monitored app should behave.

looks like something got stuck for a while then finally failed to get kicked again.

]]>https://forum.cloudron.io/post/30344https://forum.cloudron.io/post/30344Sat, 24 Apr 2021 15:55:13 GMT<![CDATA[Reply to Graphite keeps crashing OOM on Sat, 24 Apr 2021 10:12:03 GMT]]>@robi here's another one, zoomed at 24h my.armada.digital_ (1).png
Funny thing is I understand it crashes because of memory issues (resulting out of python errors?)
but why/how does Graphite reboot itself ? I mean why fail to reboot for hours and suddenly it back online? why ?

]]>https://forum.cloudron.io/post/30333https://forum.cloudron.io/post/30333Sat, 24 Apr 2021 10:12:03 GMT<![CDATA[Reply to Graphite keeps crashing OOM on Sat, 24 Apr 2021 01:41:52 GMT]]>@rmdes It's like Graphite sees Nessie the Loch Ness monster and freaks out..

Thanks for the graphs, er laughs. 😆

]]>https://forum.cloudron.io/post/30321https://forum.cloudron.io/post/30321Sat, 24 Apr 2021 01:41:52 GMT<![CDATA[Reply to Graphite keeps crashing OOM on Fri, 23 Apr 2021 18:11:37 GMT]]>my.armada.digital_.png
When graphite crash...

]]>https://forum.cloudron.io/post/30301https://forum.cloudron.io/post/30301Fri, 23 Apr 2021 18:11:37 GMT<![CDATA[Reply to Graphite keeps crashing OOM on Thu, 15 Apr 2021 20:15:02 GMT]]>I get daily crashes too, with same/similar log messages about cache and draining issues.

]]>https://forum.cloudron.io/post/29852https://forum.cloudron.io/post/29852Thu, 15 Apr 2021 20:15:02 GMT<![CDATA[Reply to Graphite keeps crashing OOM on Thu, 15 Apr 2021 11:02:48 GMT]]>@nebulon This is the only errors I find in the log, beside the restarts :
https://paste.armada.digital/xanopucuqu.sql

]]>https://forum.cloudron.io/post/29821https://forum.cloudron.io/post/29821Thu, 15 Apr 2021 11:02:48 GMT<![CDATA[Reply to Graphite keeps crashing OOM on Thu, 15 Apr 2021 08:58:10 GMT]]>All this does not sound right then. Do you see anything suspicious in the graphite logs as such? Like frequent restarts of something or so?

]]>https://forum.cloudron.io/post/29816https://forum.cloudron.io/post/29816Thu, 15 Apr 2021 08:58:10 GMT<![CDATA[Reply to Graphite keeps crashing OOM on Thu, 15 Apr 2021 08:40:11 GMT]]>@nebulon my graphite service has 1.60GB available, still OOM several times a day..
the machine where cloudron is running has 30GB available, on average 15 Gb is being used leaving half of the available memory free.

]]>https://forum.cloudron.io/post/29815https://forum.cloudron.io/post/29815Thu, 15 Apr 2021 08:40:11 GMT<![CDATA[Reply to Graphite keeps crashing OOM on Mon, 12 Apr 2021 13:42:36 GMT]]>@nebulon it was at whatever the default is (256MB?) I've now upped it to 512MB to see if that stops it. Plenty of spare RAM on the machine.

]]>https://forum.cloudron.io/post/29603https://forum.cloudron.io/post/29603Mon, 12 Apr 2021 13:42:36 GMT<![CDATA[Reply to Graphite keeps crashing OOM on Mon, 12 Apr 2021 11:58:01 GMT]]>@jdaviescoates how much memory as the limit is set in your case? Also does the server itself have enough free memory to allocate? The settings in Cloudron are only the upper limit, but it may still get killed with oom if there is none available system-wide

]]>https://forum.cloudron.io/post/29584https://forum.cloudron.io/post/29584Mon, 12 Apr 2021 11:58:01 GMT<![CDATA[Reply to Graphite keeps crashing OOM on Mon, 12 Apr 2021 09:21:25 GMT]]>Graphite OOM, again.

]]>https://forum.cloudron.io/post/29574https://forum.cloudron.io/post/29574Mon, 12 Apr 2021 09:21:25 GMT<![CDATA[Reply to Graphite keeps crashing OOM on Sat, 10 Apr 2021 13:39:11 GMT]]>I decided to reboot the box for security upgrades (from notifications) and it came up without errors this time.

]]>https://forum.cloudron.io/post/29518https://forum.cloudron.io/post/29518Sat, 10 Apr 2021 13:39:11 GMT<![CDATA[Reply to Graphite keeps crashing OOM on Sat, 10 Apr 2021 05:59:07 GMT]]>0bcb80f1-c3a8-4e0d-af61-6a02f89d7332-image.png
After a server restart, graphite won't start. Reconfig doesn't help.

]]>https://forum.cloudron.io/post/29507https://forum.cloudron.io/post/29507Sat, 10 Apr 2021 05:59:07 GMT<![CDATA[Reply to Graphite keeps crashing OOM on Fri, 09 Apr 2021 14:29:42 GMT]]>@nebulon Caprover uses Netdata... would that be possible?

]]>https://forum.cloudron.io/post/29469https://forum.cloudron.io/post/29469Fri, 09 Apr 2021 14:29:42 GMT<![CDATA[Reply to Graphite keeps crashing OOM on Fri, 09 Apr 2021 11:40:49 GMT]]>@jdaviescoates we don't know yet 😉

]]>https://forum.cloudron.io/post/29468https://forum.cloudron.io/post/29468Fri, 09 Apr 2021 11:40:49 GMT<![CDATA[Reply to Graphite keeps crashing OOM on Fri, 09 Apr 2021 11:33:49 GMT]]>Thanks

@nebulon said in Graphite keeps crashing OOM:

collecting the data on our own

What would that look like?

]]>https://forum.cloudron.io/post/29467https://forum.cloudron.io/post/29467Fri, 09 Apr 2021 11:33:49 GMT<![CDATA[Reply to Graphite keeps crashing OOM on Fri, 09 Apr 2021 11:25:22 GMT]]>@jdaviescoates that service (graphite+collectd) collects the data used in the graphs, like memory usage over time. Given that it causes issues from time to time and also we don't really utilize it well, we are thinking of maybe collecting the data on our own and ditch graphite.

]]>https://forum.cloudron.io/post/29466https://forum.cloudron.io/post/29466Fri, 09 Apr 2021 11:25:22 GMT