Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Support
  3. Graphite service orange, spinning status seen. Can't load CPU or memory graphs.

Graphite service orange, spinning status seen. Can't load CPU or memory graphs.

Scheduled Pinned Locked Moved Support
3 Posts 1 Posters 96 Views 1 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • d19dotcaD Offline
      d19dotcaD Offline
      d19dotca
      wrote on last edited by
      #1

      I've noticed the Graphite service consistently seems to spin in circles no matter how long. This prevents the CPU and memory graphs on the System page from loading correctly.

      85e4e55b-8869-488b-8046-70b92d070259-image.png

      It seems like this started happening with 8.3.1, but it's unclear if that's the case as around the same time I also migrated to a new dedicated server with the OS upgrade from 22.04 to 24.04 as well. A lot changed in a short period of time, unfortunately which makes it harder to narrow down event-wise for me.

      It normally has 512 MB assigned to the service, but I did try giving it a few GB for troubleshooting, but no difference in behaviour.

      The logs for the service after rebooting my server are included below. I noticed a connection error, perhaps this is the issue? From the log: 2025-03-27T06:09:50Z [2025-03-27T06:09:50.529Z] ERROR [tcp] read error {"error": "read tcp 172.18.0.2:2003->172.18.0.1:53106: i/o timeout"}

      Full logs below:

      2025-03-27T06:07:40Z ==> Running graphite migrations
      2025-03-27T06:07:42Z Operations to perform:
      2025-03-27T06:07:42Z Apply all migrations: account, admin, auth, contenttypes, dashboard, events, sessions, tagging, tags, url_shortener
      2025-03-27T06:07:42Z Running migrations:
      2025-03-27T06:07:42Z No migrations to apply.
      2025-03-27T06:07:42Z Your models in app(s): 'account', 'events' have changes that are not yet reflected in a migration, and so won't be applied.
      2025-03-27T06:07:42Z Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
      2025-03-27T06:07:42Z Starting graphite
      2025-03-27T06:07:42Z 2025-03-27 06:07:42,629 INFO Included extra file "/etc/supervisor/conf.d/go-carbon.conf" during parsing
      2025-03-27T06:07:42Z 2025-03-27 06:07:42,629 INFO Included extra file "/etc/supervisor/conf.d/gunicorn.conf" during parsing
      2025-03-27T06:07:42Z 2025-03-27 06:07:42,629 INFO Included extra file "/etc/supervisor/conf.d/whisper-cleanup.conf" during parsing
      2025-03-27T06:07:42Z 2025-03-27 06:07:42,630 INFO Set uid to user 0 succeeded
      2025-03-27T06:07:42Z 2025-03-27 06:07:42,639 INFO RPC interface 'supervisor' initialized
      2025-03-27T06:07:42Z 2025-03-27 06:07:42,639 CRIT Server 'inet_http_server' running without any HTTP authentication checking
      2025-03-27T06:07:42Z 2025-03-27 06:07:42,640 INFO supervisord started with pid 1
      2025-03-27T06:07:43Z 2025-03-27 06:07:43,643 INFO spawned: 'go-carbon' with pid 23
      2025-03-27T06:07:43Z 2025-03-27 06:07:43,647 INFO spawned: 'gunicorn' with pid 24
      2025-03-27T06:07:43Z 2025-03-27 06:07:43,649 INFO spawned: 'whisper-cleanup' with pid 25
      2025-03-27T06:07:43Z ==> Cleanup old whisper databases...
      2025-03-27T06:07:43Z ==> Cleanup empty directories...
      2025-03-27T06:07:43Z ==> Wait for next run in 1h...
      2025-03-27T06:07:43Z [2025-03-27T06:07:43.771Z] INFO [main] started {}
      2025-03-27T06:07:43Z [2025-03-27 06:07:43 +0000] [24] [INFO] Starting gunicorn 23.0.0
      2025-03-27T06:07:43Z [2025-03-27 06:07:43 +0000] [24] [INFO] Listening at: http://0.0.0.0:8000 (24)
      2025-03-27T06:07:43Z [2025-03-27 06:07:43 +0000] [24] [INFO] Using worker: sync
      2025-03-27T06:07:43Z [2025-03-27 06:07:43 +0000] [36] [INFO] Booting worker with pid: 36
      2025-03-27T06:07:44Z [2025-03-27 06:07:44 +0000] [37] [INFO] Booting worker with pid: 37
      2025-03-27T06:07:45Z 2025-03-27 06:07:45,057 INFO success: go-carbon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
      2025-03-27T06:07:45Z 2025-03-27 06:07:45,057 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
      2025-03-27T06:07:45Z 2025-03-27 06:07:45,057 INFO success: whisper-cleanup entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
      2025-03-27T06:08:43Z [2025-03-27T06:08:43.771Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.runtime.GOMAXPROCS", "value": 4}
      2025-03-27T06:08:43Z [2025-03-27T06:08:43.771Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.runtime.NumGoroutine", "value": 19}
      2025-03-27T06:08:43Z [2025-03-27T06:08:43.771Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.size", "value": 0}
      2025-03-27T06:08:43Z [2025-03-27T06:08:43.771Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.metrics", "value": 3}
      2025-03-27T06:08:43Z [2025-03-27T06:08:43.771Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.maxSize", "value": 1000000}
      2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.notConfirmed", "value": 4096}
      2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.queries", "value": 0}
      2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.tagsNormalizeErrors", "value": 0}
      2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.overflow", "value": 0}
      2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.queueBuildCount", "value": 598}
      2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.queueBuildTimeMs", "value": 3}
      2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.queueWriteoutTime", "value": 0}
      2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.droppedRealtimeIndex", "value": 0}
      2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.tcp.metricsReceived", "value": 761}
      2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.tcp.active", "value": 1}
      2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.tcp.errors", "value": 260}
      2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.updateOperations", "value": 761}
      2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.committedPoints", "value": 761}
      2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.pointsPerUpdate", "value": 1}
      2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.created", "value": 0}
      2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.oooDiscardedPoints", "value": 0}
      2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.maxUpdatesPerSecond", "value": 0}
      2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.workers", "value": 8}
      2025-03-27T06:08:43Z [2025-03-27T06:08:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.extended", "value": 0}
      2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.runtime.GOMAXPROCS", "value": 4}
      2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.runtime.NumGoroutine", "value": 19}
      2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.size", "value": 0}
      2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.metrics", "value": 0}
      2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.maxSize", "value": 1000000}
      2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.notConfirmed", "value": 4096}
      2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.queries", "value": 0}
      2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.tagsNormalizeErrors", "value": 0}
      2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.overflow", "value": 0}
      2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.queueBuildCount", "value": 597}
      2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.queueBuildTimeMs", "value": 0}
      2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.queueWriteoutTime", "value": 0}
      2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.droppedRealtimeIndex", "value": 0}
      2025-03-27T06:09:43Z [2025-03-27T06:09:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.tcp.metricsReceived", "value": 0}
      2025-03-27T06:09:43Z [2025-03-27T06:09:43.773Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.tcp.active", "value": 1}
      2025-03-27T06:09:43Z [2025-03-27T06:09:43.773Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.tcp.errors", "value": 0}
      2025-03-27T06:09:43Z [2025-03-27T06:09:43.773Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.updateOperations", "value": 24}
      2025-03-27T06:09:43Z [2025-03-27T06:09:43.773Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.committedPoints", "value": 24}
      2025-03-27T06:09:43Z [2025-03-27T06:09:43.773Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.pointsPerUpdate", "value": 1}
      2025-03-27T06:09:43Z [2025-03-27T06:09:43.773Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.created", "value": 0}
      2025-03-27T06:09:43Z [2025-03-27T06:09:43.773Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.oooDiscardedPoints", "value": 0}
      2025-03-27T06:09:43Z [2025-03-27T06:09:43.773Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.maxUpdatesPerSecond", "value": 0}
      2025-03-27T06:09:43Z [2025-03-27T06:09:43.773Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.workers", "value": 8}
      2025-03-27T06:09:43Z [2025-03-27T06:09:43.773Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.extended", "value": 0}
      2025-03-27T06:09:50Z [2025-03-27T06:09:50.529Z] ERROR [tcp] read error {"error": "read tcp 172.18.0.2:2003->172.18.0.1:53106: i/o timeout"}
      2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.runtime.GOMAXPROCS", "value": 4}
      2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.runtime.NumGoroutine", "value": 17}
      2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.size", "value": 0}
      2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.metrics", "value": 0}
      2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.maxSize", "value": 1000000}
      2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.notConfirmed", "value": 4096}
      2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.queries", "value": 0}
      2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.tagsNormalizeErrors", "value": 0}
      2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.overflow", "value": 0}
      2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.queueBuildCount", "value": 597}
      2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.queueBuildTimeMs", "value": 0}
      2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.queueWriteoutTime", "value": 0}
      2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.cache.droppedRealtimeIndex", "value": 0}
      2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.tcp.metricsReceived", "value": 0}
      2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.tcp.active", "value": 0}
      2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.tcp.errors", "value": 1}
      2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.updateOperations", "value": 24}
      2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.committedPoints", "value": 24}
      2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.pointsPerUpdate", "value": 1}
      2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.created", "value": 0}
      2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.oooDiscardedPoints", "value": 0}
      2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.maxUpdatesPerSecond", "value": 0}
      2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.workers", "value": 8}
      2025-03-27T06:10:43Z [2025-03-27T06:10:43.772Z] INFO [stat] collect {"endpoint": "local", "metric": "carbon.agents.graphite.persister.extended", "value": 0}
      

      --
      Dustin Dauncey
      www.d19.ca

      1 Reply Last reply
      1
      • d19dotcaD Offline
        d19dotcaD Offline
        d19dotca
        wrote on last edited by
        #2

        It seems like it's working again, but it's unclear why this is.

        --
        Dustin Dauncey
        www.d19.ca

        1 Reply Last reply
        1
        • d19dotcaD Offline
          d19dotcaD Offline
          d19dotca
          wrote on last edited by
          #3

          I wonder if this is perhaps related to the issue being troubleshot here: https://forum.cloudron.io/post/104263

          --
          Dustin Dauncey
          www.d19.ca

          1 Reply Last reply
          0
          Reply
          • Reply as topic
          Log in to reply
          • Oldest to Newest
          • Newest to Oldest
          • Most Votes


            • Login

            • Don't have an account? Register

            • Login or register to search.
            • First post
              Last post
            0
            • Categories
            • Recent
            • Tags
            • Popular
            • Bookmarks
            • Search