@girish I just restarted my server after some security updates went through and SSHFS stopped working for me as well. I deleted the backup destination and added it again and it started working. I've been on 6.3.5 for a long while (perhaps since it's release) for reference...
@humptydumpty ha ha, didn't mean to overly generalize. I think some people live by copy/paste and don't completely follow instructions, especially those who are just testing things out and want to get to it quickly.
@robi there is a already bunch of "workarounds" for rsync. Empty directories, executable bit of files cannot be stored in most object storage. So, there is fsmetadata.json file that stores this information outside of the files. When restoring, we use that file to restore back the state. I guess we can extend that file to also save and restore timestamps.
If anyone wants this leave a note and I can look into it in the future.
Funny, I just noticed this as well the other day but I figured it was just me because I'm running macOS beta so thought the new Safari overhaul killed something. haha. Glad to know it's a known issue and already fixed for the next release.
16/07/2021 17:58:24 :: [console] Error writing to collectd.localhost.df-sdc1.df_complex-used: Unable to read header (/var/lib/graphite/whisper/collectd/localhost/df-sdc1/df_complex-used.wsp)
16/07/2021 17:58:24 :: [console] Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
File "/usr/lib/python3/dist-packages/carbon/writer.py", line 189, in writeForever
--- <exception caught here> ---
File "/usr/lib/python3/dist-packages/carbon/writer.py", line 165, in writeCachedDataPoints
File "/usr/lib/python3/dist-packages/carbon/database.py", line 124, in write
File "/usr/lib/python3/dist-packages/whisper.py", line 740, in update_many
return file_update_many(fh, points, now)
File "/usr/lib/python3/dist-packages/whisper.py", line 747, in file_update_many
header = __readHeader(fh)
File "/usr/lib/python3/dist-packages/whisper.py", line 294, in __readHeader
raise CorruptWhisperFile("Unable to read header", fh.name)
whisper.CorruptWhisperFile: Unable to read header (/var/lib/graphite/whisper/collectd/localhost/df-sdc1/df_complex-free.wsp)
The last line gives a hint that the graphite file is corrupt. So, I removed all the whisper files in /home/yellowtent/platformdata/graphite/whisper/collectd/localhost/df-sdc1 and graphs seems to work after that.
@atridad Maybe a good idea to check the health of the hard disk ? Also, since we hit the systemd issue on the same server.
For future reference, systemd was acting all strange. We got all sorts of errors.
# systemctl restart box
Failed to restart box.service: Connection timed out
See system logs and 'systemctl status box.service' for details.
# systemctl status box
Failed to get properties: Connection timed out
# systemctl status box
Failed to get properties: Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)