The full stack trace is:
16/07/2021 17:58:24 :: [console] Error writing to collectd.localhost.df-sdc1.df_complex-used: Unable to read header (/var/lib/graphite/whisper/collectd/localhost/df-sdc1/df_complex-used.wsp)
16/07/2021 17:58:24 :: [console] Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
return func(*args,**kw)
File "/usr/lib/python3/dist-packages/carbon/writer.py", line 189, in writeForever
writeCachedDataPoints()
--- <exception caught here> ---
File "/usr/lib/python3/dist-packages/carbon/writer.py", line 165, in writeCachedDataPoints
state.database.write(metric, datapoints)
File "/usr/lib/python3/dist-packages/carbon/database.py", line 124, in write
whisper.update_many(path, datapoints)
File "/usr/lib/python3/dist-packages/whisper.py", line 740, in update_many
return file_update_many(fh, points, now)
File "/usr/lib/python3/dist-packages/whisper.py", line 747, in file_update_many
header = __readHeader(fh)
File "/usr/lib/python3/dist-packages/whisper.py", line 294, in __readHeader
raise CorruptWhisperFile("Unable to read header", fh.name)
whisper.CorruptWhisperFile: Unable to read header (/var/lib/graphite/whisper/collectd/localhost/df-sdc1/df_complex-free.wsp)
The last line gives a hint that the graphite file is corrupt. So, I removed all the whisper files in /home/yellowtent/platformdata/graphite/whisper/collectd/localhost/df-sdc1 and graphs seems to work after that.
@atridad Maybe a good idea to check the health of the hard disk ? Also, since we hit the systemd issue on the same server.