Cloudron not responding

[macone]

Hello
I am running Cloudron in AWS, an instance from the market. My problem is, that from time to time server stops working. On all boxes and the dashboard, I get the same error in my browser: ERR_CONNECTION_REFUSED. In this situation, I cannot connect to SSH and I have to STOP and START the server in AWS Console. This is the only thing I can do in this situation. After that everything works fine for some time.

My question is how can I diagnose what is the problem?
Thanks for any advice.

[girish]

@macone Not an expert on AWS, but many of the EC2 models (unlike lightsail) are based on the system of CPU credits. If the VM uses up the CPU "credits", the VM just stops responding. I have faced this long ago with lower EC2 models. Symptoms are similar - cannot even ssh. Do you have any monitoring/graphs for the instance in the AWS console? Can you check the CPU usage?

[macone]

@girish

[girish]

@macone Mmm, nothing obvious other than no packets are going out. Hard to know why the server stops responding entirely. Atleast, SSH should work. At the end of the day, even if Cloudron uses 100% CPU, SSH would still work (slowly). I suspect some VM issue. Maybe try to creating a snapshot (ami) and recreate the server?

[macone]

@girish I have changed EC2 type to the non-credited kind of instance. And my issue still continues.

This morning the server did not respond again and I found this is Logs:

2023-02-27T00:44:50.805Z box:apphealthmonitor app health: 15 running / 0 stopped / 0 unresponsive
Error: Something went wrong. cron reached maximum iterations.
Please open an issue (https://github.com/kelektiv/node-cron/issues/new) and provide the following string
Time Zone: "" - Cron String: 34 0,5,10,15,20,25,30,35,40,45,50,55 * * * * - UTC offset: +00:00 - current Date: Mon Feb 27 2023 05:30:05 GMT+0000
at CronTime._getNextDateFrom (/home/yellowtent/box/node_modules/cron/lib/cron.js:235:12)
at CronTime.sendAt (/home/yellowtent/box/node_modules/cron/lib/cron.js:156:17)
at CronTime.getTimeout (/home/yellowtent/box/node_modules/cron/lib/cron.js:175:29)
at CronJob.start (/home/yellowtent/box/node_modules/cron/lib/cron.js:613:31)
at Timeout.callbackWrapper [as _onTimeout] (/home/yellowtent/box/node_modules/cron/lib/cron.js:665:29)
at listOnTimeout (node:internal/timers:559:17)
at processTimers (node:internal/timers:502:7)
2023-02-27T05:42:58.021Z box:server ==========================================
2023-02-27T05:42:58.062Z box:server Cloudron 7.3.6
2023-02-27T05:42:58.062Z box:server ==========================================
2023-02-27T05:43:00.291Z box:settings initCache: pre-load settings
2023-02-27T05:43:03.531Z box:tasks stopAllTasks: stopping all tasks
2023-02-27T05:43:03.532Z box:shell stopTask spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
2023-02-27T05:43:04.569Z box:shell stopTask (stdout): sudo: unable to resolve host ip-172-30-20-28: Temporary failure in name resolution

Cloudron is up and running. Logs are at /home/yellowtent/platformdata/logs/box.log
2023-02-27T05:43:06.376Z box:shell stopTask (stdout): All tasks stopped

2023-02-27T05:43:06.393Z box:reverseproxy writeDashboardConfig: writing admin config for mydomain.net
2023-02-27T05:43:07.481Z box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
2023-02-27T05:43:07.524Z box:shell reload (stdout): sudo: unable to resolve host ip-172-30-20-28: Temporary failure in name resolution

2023-02-27T05:43:09.235Z box:shell reload (stdout): nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/_.mydomain.sk.cert" ....

[girish]

@macone said in Cloudron not responding:

Error: Something went wrong. cron reached maximum iterations.
Please open an issue (https://github.com/kelektiv/node-cron/issues/new) and provide the following string
Time Zone: "" - Cron String: 34 0,5,10,15,20,25,30,35,40,45,50,55 * * * * - UTC offset: +00:00 - current Date: Mon Feb 27 2023 05:30:05 GMT+0000

This is a bug in the upstream node module. We see this sporadically on all servers. See https://github.com/kelektiv/node-cron/issues?q=+Something+went+wrong.+cron+reached+maximum+iterations+ .. But despite it crashing, the box code will automatically restart minimizing the impact.

So, for you, it keeps crashing non-stop is it?

[macone]

@girish Yes it keeps crashing every day.

[alex-uxlabsmx]

Hello, I'm having the same issue here.

[girish]

@macone @alex-uxlabsmx If you can provide us access, we can debug the issue further. I think upstream has not solved this issue (for years) because they haven't managed to find the root cause. If you write to support@cloudron.io, we can figure out what to do next.

[macone]

@girish Thank you, I have sent the email.

[girish]

@macone Just to update you on the status, we decided to switch to another node module to avoid this bug. So, hopefully, this gets fixed in the next release.

[arshsahzad]

@macone said in Cloudron not responding:

2023-02-27T05:43:09.235Z box:shell reload (stdout): nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/_.mydomain.sk.cert" ....

Hi @girish, I'm getting the same issue in my server, earlier after reboot, It was working, but now It's not, I have raised the ticket for the same

[girish]

@arshsahzad the ocsp thing is only a warning and not a fatal error. But let's continue on support.