Cloudron instance scaling issues after a few hours / couple of days, apps responsive but showing a permanent "Starting..." status

[uwcrbc]

This has been going on for the last 3-4 weeks.

Cloudron v8.3.2 (Ubuntu 24.04 LTS)
22 Apps.

This instance have been running fine for a couple of years at least.
On a fresh start or reboot, the instance works just fine.
However with time going by, with no changes of significance and no out of the ordinary operations (use of the apps, scheduled(successful) daily backup, automatic app updates)

With 1-2 days the instance show signs of a "strange" nature:

• App restart (after an update for example, but this goes with manual restart too) show the app locked into a "Starting..." state on the dashboard.
• The app itself is most of the time still responsive and can be used as normal.
• As far as I can tell, both the "standard" logs of the app as well as the log of the Cloudron instance are not showing anything of relevance / out of the ordinary
• Performance indicators from the Cloudron instance seem to be perfectly fine.

More time goes, more apps show the same "Starting...." state.
It does not matter which app it seems.

If left untouched, other problems appears on the Cloudron instance such as email fetching problem without a Freescout app for example.

A quick fix, but not a permanent one is to run systemctl restart box on the instance terminal.
This returns the apps status back to "Running" and things like the fetching of email within Freescout is back to operational.

Also running cloudron-support --troubleshoot return all green / does not show any error.

Has someone been across this behavior already? What would be the next step in troubleshooting and/or permanently solving this?

[james]

Hello @uwcrbc
Can you please run cloudron-support --send-diagnostics on root and provide me the generated link?
Since the link includes some sensitive data like your Cloudron URL you can send that link to me in a private message if you'd like.

[uwcrbc]

@james Hi James - I have now sent you the link. Thanks.

[james]

I have reviewed the data and could find anything really suspicious.

[uwcrbc]

@james said in Cloudron instance scaling issues after a few hours / couple of days, apps responsive but showing a permanent "Starting..." status:

I have reviewed the data and could find anything really suspicious.

"couldn't find....." ?

How do we go from there? Happy to enable remote SSH support if helpful.

Many thanks,

[nebulon]

@uwcrbc please send a mail with your Cloudron ID / dashbaord domain info to support@cloudron.io and enable remote ssh support for us, so we can take a closer look to get this resolved.

[uwcrbc]

@nebulon Hi @nebulon - Thanks for this. Email sent.

[nebulon]

We got the info and are looking into this. The findings so far are, that the calcom app is constantly restarting and causing the server to become very slow. Not sure why this seems to have cascading effects, since the resources are plenty, but depending on the server provider, the virtual CPUs might get throttled if one core is used 100% over a longer period of time. Will keep this updated on the findings.

[uwcrbc]

@nebulon Interesting.
The server provider is ssdnodes - So far I have had nothing to complain about (just about the opposite). Happy to play the liaison part should we need to relay info / ask questions of them.
Many thanks,

[uwcrbc]

Also: interestingly the moment the server started playing up is indeed the moment that I had roughly linked to our latest cal.com deployment.
I say latest because this Cloudron server had a previous cal.com app instance that we used as a test a while back - not sure if relevant but maybe it can help inform the situation.

[nebulon]

I've updated the calcom instance on that server to latest, which has some fixes in the package. So far things seem fine.