Graphite keeps crashing OOM
-
Graphite OOM, again.
@jdaviescoates how much memory as the limit is set in your case? Also does the server itself have enough free memory to allocate? The settings in Cloudron are only the upper limit, but it may still get killed with oom if there is none available system-wide
-
@jdaviescoates how much memory as the limit is set in your case? Also does the server itself have enough free memory to allocate? The settings in Cloudron are only the upper limit, but it may still get killed with oom if there is none available system-wide
@nebulon it was at whatever the default is (256MB?) I've now upped it to 512MB to see if that stops it. Plenty of spare RAM on the machine.
-
@jdaviescoates how much memory as the limit is set in your case? Also does the server itself have enough free memory to allocate? The settings in Cloudron are only the upper limit, but it may still get killed with oom if there is none available system-wide
-
All this does not sound right then. Do you see anything suspicious in the graphite logs as such? Like frequent restarts of something or so?
-
@nebulon This is the only errors I find in the log, beside the restarts :
https://paste.armada.digital/xanopucuqu.sql -
@nebulon my graphite service has 1.60GB available, still OOM several times a day..
the machine where cloudron is running has 30GB available, on average 15 Gb is being used leaving half of the available memory free. -
@rmdes It's like Graphite sees Nessie the Loch Ness monster and freaks out..
Thanks for the graphs, er laughs.
@robi here's another one, zoomed at 24h
Funny thing is I understand it crashes because of memory issues (resulting out of python errors?)
but why/how does Graphite reboot itself ? I mean why fail to reboot for hours and suddenly it back online? why ? -
@robi here's another one, zoomed at 24h
Funny thing is I understand it crashes because of memory issues (resulting out of python errors?)
but why/how does Graphite reboot itself ? I mean why fail to reboot for hours and suddenly it back online? why ? -
Maybe this python error can help ? https://paste.armada.digital/ovurasajof.sql
-
@rmdes are you able to write to me on support@ and give me ssh access, so I can debug this? Would be good understand what's happening here.
-
@rmdes thanks for the access. it seems your server somehow hits this carbon cache bug - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=923464
-
-
@rmdes I have applied the patch in the bug report and it seems to fix the problem. I have applied change to your server locally. Will be in next release.
-
@girish So this only hit on me ?
Anyway, Thanks a lot for applying the patch locally and fixing the issue !@rmdes yes, I am not sure why. It doesn't happen in any of our demo servers or managed services. Quite strange. It could also be that maybe others have hit it but have not noticed it (since it only causes a CPU spike..) but clearly it's a bug since it's been fixed upstream.
