Orange circles in Services

[subven]

Hmm I have a similar issue on a Cloudron (v6.1.2 and before) where random apps are stuck in "Starting" but apparently working/running as they should. Services however work just fine on this Cloudron.

This state is kinda permanent and restarting the app nor the server does not help. Also seems to be kinda random what app is affected but their numbers seem to grow. My other two Cloudrons are fine ^^

Log output as en example (for Element) is:

Feb 16 20:36:35 box:settings initCache: pre-load settings
Feb 16 20:36:35 box:taskworker Starting task 789. Logs are at /home/yellowtent/platformdata/logs/e0016eed-67a4-45dd-8234-642c8aee11d3/apptask.log
Feb 16 20:36:35 box:tasks 789: {"percent":2,"error":null}
Feb 16 20:36:35 box:apptask chat.heregoesthedomain.com startTask installationState: pending_restart runState: running
Feb 16 20:36:35 box:tasks 789: {"percent":20,"message":"Restarting container"}
Feb 16 20:36:35 http-server stopped.
Feb 16 20:36:36 ==> Starting Riot
Feb 16 20:36:36 box:tasks 789: {"percent":100,"message":"Done"}
Feb 16 20:36:36 box:apptask chat.heregoesthedomain.com updating app with values: {"installationState":"installed","error":null,"health":null}
Feb 16 20:36:36 box:taskworker Task took 1.109 seconds
Feb 16 20:36:36 box:tasks setCompleted - 789: {"result":null,"error":null}
Feb 16 20:36:36 box:tasks 789: {"percent":100,"result":null,"error":null}
Feb 16 20:36:36 Starting up http-server, serving /app/code
Feb 16 20:36:36 Available on:
Feb 16 20:36:36 http://127.0.0.1:3000
Feb 16 20:36:36 http://172.18.16.9:3000
Feb 16 20:36:36 Hit CTRL-C to stop the server
Feb 16 20:36:57 [2021-02-16T19:36:57.389Z] "GET /" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36"

PS: Sorry, not wanted to hijack this topic. I thought about opening an seperate issue for this but so far it was not that much of a problem and I hoped it would be fixed after the next update ^^

[girish]

@subven I think the starting issue is something else. If you do systemctl restart box, I think they will all become running again. Can you please try?

[subven]

@girish said in Orange circles in Services:

@subven I think the starting issue is something else. If you do systemctl restart box, I think they will all become running again. Can you please try?

Yes they are I was confused because even box:apphealthmonitor said the apps are up.

[robi]

something is getting lost between the checks and the UI representation.

[d19dotca]

This appears to still be an issue on 6.2.1 by the way, though I seem to get this mostly with Mail now more than the others.

[subven]

Kinda same for me. Freshly installed server + imported Cloudron 6.1.2 backup. The server the backup is from did not had the problem.

[nebulon]

@subven interesting that it shows an "active" tooltip. Is this state consistent across page reloads?

[subven]

@nebulon said in Orange circles in Services:

@subven interesting that it shows an "active" tooltip. Is this state consistent across page reloads?

Mostly yes, red circles vary after reload. Seems to be my local Chrome because it works with FF and on other machines with Chrome. Strg+F5 and deleting the local site data storage does not help. It's fine for me since its only cosmetic.

[nebulon]

@subven I think this is some client side state race or so. Any information here in behavior can help

[subven]

@nebulon I solved it. It was caused by one of my chrome addons (Dark Reader)

[robi]

@subven can you be more specific?

[subven]

@robi regarding to what? I had a similiar issue than other users (multiple false positive red circles) and I fixed it by disabling a Chrome addon (Dark Reader) that alters page CSS. I can reproduce it by enabling the addon for the dashboard domain.

Dark Reader turned on:

Dark Reader turned off:

[robi]

@subven thanks, thought perhaps you modified a setting within DR

[d19dotca]

@subven I think that may be specific to your use-case but it's still good to know it's fixed for you. I'm using Safari and have no extensions installed other than a password manager. Nothing that interferes with the webpage content. I've disabled some settings too as a result though to test further (like Safari's non-tracking features and stuff) but no difference in my case. I also use Mobile Safari where I see the issue and that has zero extensions on it at all.

I could be wrong, but I think the issue started around the time 6.1.2 came out. I'm not convinced this is a browser issue, although I think that's what @nebulon is suggesting if I understand him correctly.

@robi - in case this is indeed a browser / client-side issue (which I don't think it is personally), maybe note which browser you're using and accessing Cloudron on where you see this issue too in case it's Safari for both you and me. I suspect the behaviour @subven has seen is entirely separate from what you and I are reporting, judging by their comment on Feb 16th although admittedly similar in aspects.

[nebulon]

@d19dotca I also think that maybe that darkmode plugin just changes the timing and does not cause the race to hit badly.

[robi]

@nebulon for me, this still happens in a non dark mode enabled chrome based browser.

[girish]

@robi Is this the same server that had the etherpad issue? Because that one seems just fine here. I tried refreshing many times now, always green.

[necrevistonnezr]

@girish
I, too, have several services from time to time indicating that they are starting though they seem to be running fine - such as mail in this example.

[nebulon]

This is a very curious case. The tooltip of that status indicator is using the exact same property as the color. Both are not updated individually. So far I was not able to see how there could be some states out-of-sync if status is active.

I do see from the screenshots that the restart button also spins though, this indicates in some of those screenshots that the status is indeed starting not active for which the UI then seems to be consistent.

This leads to a different question, if the service is actually running fine, why does the REST api then report starting? From the code, this only happens if the upstream service instance (they live in their own container) fails to report status, while the container being up.

Now to get more information, it would help if some of you, who can reproduce this, could take a look at the box logs at /home/yellowtent/platformdata/logs/box.log while refreshing the dashboard. This should trigger a call to all the service containers and thus should hopefully give us more information. For example the call might just timeout or otherwise fail.

[robi]

@nebulon After waking up a sleeping tab, I had graphite and mongo in orange. Graphite was spinning.

After reloading the services page, mongo went green. I then edited the graphite service and just saved. It then went green too.

these are the box logs.

2021-03-09T16:50:21.009Z box:translation Languages found: [ 'de.json', 'en.json', 'it.json', 'nl.json', 'zh_Hans.json' ]
2021-03-09T16:50:21.332Z box:apphealthmonitor app health: 21 alive / 1 dead.
2021-03-09T16:50:23.093Z box:shell statusUnbound exec: systemctl is-active unbound
2021-03-09T16:50:23.146Z box:shell statusNginx exec: systemctl is-active nginx
2021-03-09T16:50:23.178Z box:shell statusUnbound (stdout): active

2021-03-09T16:50:23.178Z box:shell statusUnbound (stderr):
2021-03-09T16:50:23.193Z box:shell statusNginx (stdout): active

2021-03-09T16:50:23.193Z box:shell statusNginx (stderr):
2021-03-09T16:50:31.262Z box:apphealthmonitor app health: 21 alive / 1 dead.
2021-03-09T16:50:41.027Z box:apphealthmonitor app health: 21 alive / 1 dead.
2021-03-09T16:50:50.875Z box:apphealthmonitor app health: 21 alive / 1 dead.
2021-03-09T16:50:59.971Z box:addons updateServiceConfig: graphite {"memoryLimit":268435456}
2021-03-09T16:51:00.038Z box:shell update(graphite) spawn: /usr/bin/docker update --memory 241172480 --memory-swap 268435456 graphite
2021-03-09T16:51:00.735Z box:apphealthmonitor app health: 21 alive / 1 dead.
2021-03-09T16:51:01.190Z box:shell update(graphite) (stdout): graphite

2021-03-09T16:51:10.840Z box:apphealthmonitor app health: 21 alive / 1 dead.