Cloudron instance scaling issues after a few hours / couple of days, apps responsive but showing a permanent "Starting..." status

[uwcrbc]

@nebulon Interesting.
The server provider is ssdnodes - So far I have had nothing to complain about (just about the opposite). Happy to play the liaison part should we need to relay info / ask questions of them.
Many thanks,

[uwcrbc]

Also: interestingly the moment the server started playing up is indeed the moment that I had roughly linked to our latest cal.com deployment.
I say latest because this Cloudron server had a previous cal.com app instance that we used as a test a while back - not sure if relevant but maybe it can help inform the situation.

[nebulon]

I've updated the calcom instance on that server to latest, which has some fixes in the package. So far things seem fine.

[uwcrbc]

Happy to report that the updated cal.com package seems to have fixed the underlying server issue.
Many thanks,

[uwcrbc]

At last - I think I was too eager to see this resolved.
The problem is still there with the exact same symptoms.
How can we proceed further? @nebulon @james Would another remote troubleshooting/debugging be helpful?

[nebulon]

Do you see a constant process showing up in htop or so on your system, which may constantly hog some core like calcom restart did?

[uwcrbc]

@nebulon Thanks - To the untrained eye, there is nothing in htop that pops up as flagrantly hogging resources.
The swap is high but I am not sure if this is a relevant indicator

[nebulon]

And the systemctl restart box still fixes the issue temporarily?

[uwcrbc]

@nebulon Yes - this fixes the issue temporarily at least

[girish]

This is probably happening because the apphealthmonitor is getting stuck (for some reason) and thus unable to update the health of apps. @uwcrbc in the box.log , do you see a line printed every 10 seconds like box:apphealthmonitor app health: xx ?

[uwcrbc]

@girish Thanks for this - After looking into the box.log:

• post systemctl restart box I do see indeed box:apphealthmonitor app health: xx running / 0 stopped / 0 unresponsive entry type every 10 seconds or so.
• pre systemctl restart box (when we experience the issue) I do not see much of the box:apphealthmonitor app health:xx entries. Rather, I do have a few rare box:apphealthmonitor setHealth: <<CONTAINER_UID>> (<<URL>>) waiting for 1192.461 to update health entries

Hopefully it helps?

[girish]

@uwcrbc I think there is a bug that the apphealthmonitor is getting stuck (for some reason). We have seen this happen but very rarely and not reproducibly . If you can give us access, can you send me a mail at support@cloudron.io ? I can debug this further .

[Neiluj]

This post is deleted!

[uwcrbc]

@girish Thanks for this. Access given / mail sent. Appreciate the help.