Graphite service orange, spinning status seen. Can't load CPU or memory graphs.

[d19dotca]

I've noticed the Graphite service consistently seems to spin in circles no matter how long. This prevents the CPU and memory graphs on the System page from loading correctly.

It seems like this started happening with 8.3.1, but it's unclear if that's the case as around the same time I also migrated to a new dedicated server with the OS upgrade from 22.04 to 24.04 as well. A lot changed in a short period of time, unfortunately which makes it harder to narrow down event-wise for me.

It normally has 512 MB assigned to the service, but I did try giving it a few GB for troubleshooting, but no difference in behaviour.

The logs for the service after rebooting my server are included below. I noticed a connection error, perhaps this is the issue? From the log: 2025-03-27T06:09:50Z [2025-03-27T06:09:50.529Z] ERROR [tcp] read error {"error": "read tcp 172.18.0.2:2003->172.18.0.1:53106: i/o timeout"}

Full logs below:

2025-03-27T06:07:40Z ==> Running graphite migrations
2025-03-27T06:07:42Z Operations to perform:
2025-03-27T06:07:42Z Apply all migrations: account, admin, auth, contenttypes, dashboard, events, sessions, tagging, tags, url_shortener
2025-03-27T06:07:42Z Running migrations:
2025-03-27T06:07:42Z No migrations to apply.
2025-03-27T06:07:42Z Your models in app(s): 'account', 'events' have changes that are not yet reflected in a migration, and so won't be applied.
2025-03-27T06:07:42Z Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
2025-03-27T06:07:42Z Starting graphite
2025-03-27T06:07:42Z 2025-03-27 06:07:42,629 INFO Included extra file "/etc/supervisor/conf.d/go-carbon.conf" during parsing
2025-03-27T06:07:42Z 2025-03-27 06:07:42,629 INFO Included extra file "/etc/supervisor/conf.d/gunicorn.conf" during parsing
2025-03-27T06:07:42Z 2025-03-27 06:07:42,629 INFO Included extra file "/etc/supervisor/conf.d/whisper-cleanup.conf" during parsing
2025-03-27T06:07:42Z 2025-03-27 06:07:42,630 INFO Set uid to user 0 succeeded
2025-03-27T06:07:42Z 2025-03-27 06:07:42,639 INFO RPC interface 'supervisor' initialized
2025-03-27T06:07:42Z 2025-03-27 06:07:42,639 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2025-03-27T06:07:42Z 2025-03-27 06:07:42,640 INFO supervisord started with pid 1
2025-03-27T06:07:43Z 2025-03-27 06:07:43,643 INFO spawned: 'go-carbon' with pid 23
2025-03-27T06:07:43Z 2025-03-27 06:07:43,647 INFO spawned: 'gunicorn' with pid 24
2025-03-27T06:07:43Z 2025-03-27 06:07:43,649 INFO spawned: 'whisper-cleanup' with pid 25
2025-03-27T06:07:43Z ==> Cleanup old whisper databases...
2025-03-27T06:07:43Z ==> Cleanup empty directories...
2025-03-27T06:07:43Z ==> Wait for next run in 1h...
2025-03-27T06:07:43Z [2025-03-27T06:07:43.771Z] INFO [main] started {}
2025-03-27T06:07:43Z [2025-03-27 06:07:43 +0000] [24] [INFO] Starting gunicorn 23.0.0
2025-03-27T06:07:43Z [2025-03-27 06:07:43 +0000] [24] [INFO] Listening at: http://0.0.0.0:8000 (24)
2025-03-27T06:07:43Z [2025-03-27 06:07:43 +0000] [24] [INFO] Using worker: sync
2025-03-27T06:07:43Z [2025-03-27 06:07:43 +0000] [36] [INFO] Booting worker with pid: 36
2025-03-27T06:07:44Z [2025-03-27 06:07:44 +0000] [37] [INFO] Booting worker with pid: 37
2025-03-27T06:07:45Z 2025-03-27 06:07:45,057 INFO success: go-carbon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-03-27T06:07:45Z 2025-03-27 06:07:45,057 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-03-27T06:07:45Z 2025-03-27 06:07:45,057 INFO success: whisper-cleanup entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-03-27T06:08:43Z [2025-03-27T06:08:43.771Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.runtime.GOMAXPROCS", "value": 4}
2025-03-27T06:08:43Z [2025-03-27T06:08:43.771Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.runtime.NumGoroutine", "value": 19}
2025-03-27T06:08:43Z [2025-03-27T06:08:43.771Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.size", "value": 0}
2025-03-27T06:08:43Z [2025-03-27T06:08:43.771Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.metrics", "value": 3}
2025-03-27T06:08:43Z [2025-03-27T06:08:43.771Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.maxSize", "value": 1000000}
2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.notConfirmed", "value": 4096}
2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.queries", "value": 0}
2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.tagsNormalizeErrors", "value": 0}
2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.overflow", "value": 0}
2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.queueBuildCount", "value": 598}
2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.queueBuildTimeMs", "value": 3}
2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.queueWriteoutTime", "value": 0}
2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.droppedRealtimeIndex", "value": 0}
2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.tcp.metricsReceived", "value": 761}
2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.tcp.active", "value": 1}
2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.tcp.errors", "value": 260}
2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.updateOperations", "value": 761}
2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.committedPoints", "value": 761}
2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.pointsPerUpdate", "value": 1}
2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.created", "value": 0}
2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.oooDiscardedPoints", "value": 0}
2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.maxUpdatesPerSecond", "value": 0}
2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.workers", "value": 8}
2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.extended", "value": 0}
2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.runtime.GOMAXPROCS", "value": 4}
2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.runtime.NumGoroutine", "value": 19}
2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.size", "value": 0}
2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.metrics", "value": 0}
2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.maxSize", "value": 1000000}
2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.notConfirmed", "value": 4096}
2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.queries", "value": 0}
2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.tagsNormalizeErrors", "value": 0}
2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.overflow", "value": 0}
2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.queueBuildCount", "value": 597}
2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.queueBuildTimeMs", "value": 0}
2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.queueWriteoutTime", "value": 0}
2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.droppedRealtimeIndex", "value": 0}
2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.tcp.metricsReceived", "value": 0}
2025-03-27T06:09:43Z [2025-03-27T06:09:43.773Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.tcp.active", "value": 1}
2025-03-27T06:09:43Z [2025-03-27T06:09:43.773Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.tcp.errors", "value": 0}
2025-03-27T06:09:43Z [2025-03-27T06:09:43.773Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.updateOperations", "value": 24}
2025-03-27T06:09:43Z [2025-03-27T06:09:43.773Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.committedPoints", "value": 24}
2025-03-27T06:09:43Z [2025-03-27T06:09:43.773Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.pointsPerUpdate", "value": 1}
2025-03-27T06:09:43Z [2025-03-27T06:09:43.773Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.created", "value": 0}
2025-03-27T06:09:43Z [2025-03-27T06:09:43.773Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.oooDiscardedPoints", "value": 0}
2025-03-27T06:09:43Z [2025-03-27T06:09:43.773Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.maxUpdatesPerSecond", "value": 0}
2025-03-27T06:09:43Z [2025-03-27T06:09:43.773Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.workers", "value": 8}
2025-03-27T06:09:43Z [2025-03-27T06:09:43.773Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.extended", "value": 0}
2025-03-27T06:09:50Z [2025-03-27T06:09:50.529Z] ERROR [tcp] read error {"error": "read tcp 172.18.0.2:2003->172.18.0.1:53106: i/o timeout"}
2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.runtime.GOMAXPROCS", "value": 4}
2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.runtime.NumGoroutine", "value": 17}
2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.size", "value": 0}
2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.metrics", "value": 0}
2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.maxSize", "value": 1000000}
2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.notConfirmed", "value": 4096}
2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.queries", "value": 0}
2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.tagsNormalizeErrors", "value": 0}
2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.overflow", "value": 0}
2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.queueBuildCount", "value": 597}
2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.queueBuildTimeMs", "value": 0}
2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.queueWriteoutTime", "value": 0}
2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.droppedRealtimeIndex", "value": 0}
2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.tcp.metricsReceived", "value": 0}
2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.tcp.active", "value": 0}
2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.tcp.errors", "value": 1}
2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.updateOperations", "value": 24}
2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.committedPoints", "value": 24}
2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.pointsPerUpdate", "value": 1}
2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.created", "value": 0}
2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.oooDiscardedPoints", "value": 0}
2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.maxUpdatesPerSecond", "value": 0}
2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.workers", "value": 8}
2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.extended", "value": 0}