Box Crashes.

roofboard

From time to time cloudron crashes.

All services stop working, and when the my.domain.com finally loads it looks just like this.

and when I log into cloudron I can 'fix' it by restarting box, then everything goes back to normal.

root@my:~# systemctl status nginx
● nginx.service - nginx - high performance web server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/nginx.service.d
             └─cloudron.conf
     Active: active (running) since Fri 2022-11-18 02:01:06 UTC; 3 weeks 6 days ago
       Docs: http://nginx.org/en/docs/
   Main PID: 5728 (nginx)
      Tasks: 7 (limit: 28752)
     Memory: 46.8M
     CGroup: /system.slice/nginx.service
             ├─  5728 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
             ├─424780 nginx: worker process
             ├─424781 nginx: worker process
             ├─424782 nginx: worker process
             ├─424783 nginx: worker process
             ├─424784 nginx: worker process
             └─424785 nginx: worker process

Warning: journal has been rotated since unit was started, output may be incomplete.
root@my:~# systemctl status box
● box.service - Cloudron Admin
     Loaded: loaded (/etc/systemd/system/box.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2022-12-05 09:14:04 UTC; 1 weeks 3 days ago
    Process: 501731 ExecReload=/usr/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
   Main PID: 3006796 (node)
      Tasks: 11 (limit: 28752)
     Memory: 100.8M (max: 400.0M)
     CGroup: /system.slice/box.service
             └─3006796 node /home/yellowtent/box/box.js

Dec 15 03:23:00 my.domain.com sudo[691844]: pam_unix(sudo:session): session opened for user root by (uid=0)
Dec 15 03:24:05 my.domain.com sudo[691844]: pam_unix(sudo:session): session closed for user root
Dec 15 03:30:00 my.domain.com sudo[696164]: pam_unix(sudo:session): session opened for user root by (uid=0)
Dec 15 03:34:23 my.domain.com sudo[696164]: pam_unix(sudo:session): session closed for user root
Dec 15 05:30:00 my.domain.com sudo[768909]: pam_unix(sudo:session): session opened for user root by (uid=0)
Dec 15 05:35:34 my.domain.com sudo[768909]: pam_unix(sudo:session): session closed for user root
Dec 15 07:00:05 my.domain.com sudo[823705]: pam_unix(sudo:session): session opened for user root by (uid=0)
Dec 15 07:04:32 my.domain.com sudo[823705]: pam_unix(sudo:session): session closed for user root
Warning: journal has been rotated since unit was started, output may be incomplete.
root@my:~# systemctl status unbound
● unbound.service - Unbound DNS Resolver
     Loaded: loaded (/etc/systemd/system/unbound.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2022-12-05 09:14:03 UTC; 1 weeks 3 days ago
   Main PID: 3006652 (unbound)
      Tasks: 1 (limit: 28752)
     Memory: 21.8M
     CGroup: /system.slice/unbound.service
             └─3006652 /usr/sbin/unbound -d

Dec 15 15:07:41 my.domain.com unbound[3006652]: [3006652:0] info: generate keytag query _ta-4f66. NULL IN
Dec 15 15:12:52 my.domain.com unbound[3006652]: [3006652:0] info: generate keytag query _ta-4f66. NULL IN
Dec 15 15:18:01 my.domain.com unbound[3006652]: [3006652:0] info: generate keytag query _ta-4f66. NULL IN
Dec 15 15:23:12 my.domain.com unbound[3006652]: [3006652:0] info: generate keytag query _ta-4f66. NULL IN
Dec 15 15:28:27 my.domain.com unbound[3006652]: [3006652:0] info: generate keytag query _ta-4f66. NULL IN
Dec 15 15:33:39 my.domain.com unbound[3006652]: [3006652:0] info: generate keytag query _ta-4f66. NULL IN
Dec 15 15:38:52 my.domain.com unbound[3006652]: [3006652:0] info: generate keytag query _ta-4f66. NULL IN
Dec 15 15:44:05 my.domain.com unbound[3006652]: [3006652:0] info: generate keytag query _ta-4f66. NULL IN
Dec 15 15:49:17 my.domain.com unbound[3006652]: [3006652:0] info: generate keytag query _ta-4f66. NULL IN
Dec 15 15:54:30 my.domain.com unbound[3006652]: [3006652:0] info: generate keytag query _ta-4f66. NULL IN
root@my:~# systemctl restart box
root@my:~# systemctl status box
● box.service - Cloudron Admin
     Loaded: loaded (/etc/systemd/system/box.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-12-15 16:01:45 UTC; 10min ago
   Main PID: 1152712 (node)
      Tasks: 11 (limit: 28752)
     Memory: 82.9M (max: 400.0M)
     CGroup: /system.slice/box.service
             └─1152712 node /home/yellowtent/box/box.js

Dec 15 16:01:45 my.domain.com systemd[1]: Started Cloudron Admin.
Dec 15 16:01:47 my.domain.com sudo[1152769]: pam_unix(sudo:session): session opened for user root by (uid=0)
Dec 15 16:01:47 my.domain.com sudo[1152769]: pam_unix(sudo:session): session closed for user root
Dec 15 16:01:47 my.domain.com sudo[1152777]: pam_unix(sudo:session): session opened for user root by (uid=0)
Dec 15 16:01:57 my.domain.com sudo[1152777]: pam_unix(sudo:session): session closed for user root
Dec 15 16:02:27 my.domain.com sudo[1153127]: pam_unix(sudo:session): session opened for user root by (uid=0)
Dec 15 16:02:53 my.domain.com sudo[1153127]: pam_unix(sudo:session): session closed for user root
root@my:~#

nebulon

Do you see anything in the logs at /home/yellowtent/platformdata/logs/box.log ?

roofboard

@nebulon here are the logs leading up to rebooting the server - Note when i restarted box it helped only temporarily but when i reboot the server i get much more uptime.

there are some time gaps because i had to truncate out noise to get under the char limit


2022-12-15T16:15:40.145Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:15:50.034Z box:apphealthmonitor setHealth: f30ffad7-b996-4ecc-a6cf-c375ab916f19 (help.troubled-nodebb-installation.com) waiting for 369.967 to update health
2022-12-15T16:15:50.170Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:16:00.051Z box:apphealthmonitor setHealth: f30ffad7-b996-4ecc-a6cf-c375ab916f19 (help.troubled-nodebb-installation.com) waiting for 359.95 to update health
2022-12-15T16:16:00.231Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:16:10.052Z box:apphealthmonitor setHealth: f30ffad7-b996-4ecc-a6cf-c375ab916f19 (help.troubled-nodebb-installation.com) waiting for 349.949 to update health
2022-12-15T16:16:10.209Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:16:11.595Z box:shell statusUnbound exec: systemctl is-active unbound
2022-12-15T16:16:11.615Z box:shell statusUnbound (stdout): active

2022-12-15T16:16:11.615Z box:shell statusUnbound (stderr): null
2022-12-15T16:16:11.668Z box:shell statusNginx exec: systemctl is-active nginx
2022-12-15T16:16:11.689Z box:shell statusNginx (stdout): active

2022-12-15T16:16:11.689Z box:shell statusNginx (stderr): null
2022-12-15T16:16:20.025Z box:apphealthmonitor setHealth: f30ffad7-b996-4ecc-a6cf-c375ab916f19 (help.troubled-nodebb-installation.com) waiting for 339.976 to update health
2022-12-15T16:16:20.155Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:16:20.674Z box:cloudron Getting logs for box as json
2022-12-15T16:16:30.036Z box:apphealthmonitor setHealth: f30ffad7-b996-4ecc-a6cf-c375ab916f19 (help.troubled-nodebb-installation.com) waiting for 329.965 to update health
2022-12-15T16:20:20.152Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:20:30.037Z box:apphealthmonitor setHealth: f30ffad7-b996-4ecc-a6cf-c375ab916f19 (help.troubled-nodebb-installation.com) waiting for 89.964 to update health
2022-12-15T16:20:30.194Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:20:40.040Z box:apphealthmonitor setHealth: f30ffad7-b996-4ecc-a6cf-c375ab916f19 (help.troubled-nodebb-installation.com) waiting for 79.963 to update health
2022-12-15T16:20:40.136Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:20:50.037Z box:apphealthmonitor setHealth: f30ffad7-b996-4ecc-a6cf-c375ab916f19 (help.troubled-nodebb-installation.com) waiting for 69.964 to update health
2022-12-15T16:20:50.164Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:21:00.048Z box:apphealthmonitor setHealth: f30ffad7-b996-4ecc-a6cf-c375ab916f19 (help.troubled-nodebb-installation.com) waiting for 59.953 to update health
2022-12-15T16:21:00.196Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:21:10.460Z box:apphealthmonitor setHealth: f30ffad7-b996-4ecc-a6cf-c375ab916f19 (help.troubled-nodebb-installation.com) waiting for 49.541 to update health
2022-12-15T16:21:10.460Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:21:20.045Z box:apphealthmonitor setHealth: f30ffad7-b996-4ecc-a6cf-c375ab916f19 (help.troubled-nodebb-installation.com) waiting for 39.956 to update health
2022-12-15T16:21:20.211Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:21:30.036Z box:apphealthmonitor setHealth: f30ffad7-b996-4ecc-a6cf-c375ab916f19 (help.troubled-nodebb-installation.com) waiting for 29.965 to update health
2022-12-15T16:21:30.178Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:21:40.039Z box:apphealthmonitor setHealth: f30ffad7-b996-4ecc-a6cf-c375ab916f19 (help.troubled-nodebb-installation.com) waiting for 19.962 to update health
2022-12-15T16:21:40.179Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:21:50.031Z box:apphealthmonitor setHealth: f30ffad7-b996-4ecc-a6cf-c375ab916f19 (help.troubled-nodebb-installation.com) waiting for 9.97 to update health
2022-12-15T16:21:50.202Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:22:00.175Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:22:10.175Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:22:20.175Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:29:30.233Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:29:40.147Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:29:50.245Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:30:00.007Z box:disks checkDiskSpace: checking disk space
2022-12-15T16:30:00.059Z box:janitor Cleaning up expired tokens
2022-12-15T16:30:00.065Z box:janitor Cleaned up 0 expired tokens
2022-12-15T16:30:00.432Z box:disks checkDiskSpace: disk space checked. out of space: no
2022-12-15T16:30:00.450Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:30:10.402Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:30:20.150Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:30:30.170Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:30:40.184Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:30:50.194Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:31:00.163Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:31:10.185Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T16:31:20.140Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:32:40.168Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:32:50.121Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:33:00.191Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:33:10.171Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:33:21.168Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:33:30.173Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:33:40.164Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:33:50.164Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:34:00.232Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:34:10.194Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:34:20.199Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:34:30.163Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:34:40.142Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:34:50.194Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:35:00.211Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:35:10.115Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:35:20.176Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:35:30.167Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:35:41.721Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:35:50.166Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:36:00.155Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:36:10.201Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:36:20.220Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:36:30.175Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:36:40.187Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:36:50.222Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:37:00.148Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:37:10.219Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:37:20.148Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:37:30.151Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:37:40.217Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:37:50.184Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:38:00.127Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:38:10.180Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:38:20.131Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:38:30.158Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:38:40.128Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:38:50.155Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:39:00.214Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:39:10.203Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:39:20.135Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:39:30.184Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:39:40.206Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:39:50.159Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:40:00.203Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:40:10.169Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:40:20.129Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:40:30.214Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:40:40.165Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:40:50.147Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:41:00.182Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:41:10.193Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:41:21.093Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:41:30.212Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:41:40.137Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:41:50.202Z box:apphealthmonitor app health: 15 running / 2 stopped / 1 unresponsive
2022-12-15T17:41:55.205Z box:box Received SIGTERM. Shutting down.
2022-12-15T17:41:55.207Z box:tasks stopAllTasks: stopping all tasks
2022-12-15T17:41:55.208Z box:shell stopTask spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
2022-12-15T17:41:58.578Z box:cloudron Getting logs for box as json
2022-12-15T17:43:00.113Z box:server ==========================================
2022-12-15T17:43:00.127Z box:server            Cloudron 7.3.4  
2022-12-15T17:43:00.127Z box:server ==========================================
2022-12-15T17:43:00.573Z box:settings initCache: pre-load settings
2022-12-15T17:43:00.635Z box:tasks stopAllTasks: stopping all tasks
2022-12-15T17:43:00.636Z box:shell stopTask spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
Cloudron is up and running. Logs are at /home/yellowtent/platformdata/logs/box.log
2022-12-15T17:43:00.925Z box:shell stopTask (stdout): All tasks stopped

nebulon

There is really nothing much in the logs which might indicate an issue. Also from your initial screenshot it looks like the static assets served up by nginx are timing out. If this happens again, can you see if restart nginx helps as well?

roofboard

@nebulon
this problem has been ongoing for a number of weeks server goes down and scheduling a downdetector to restart Cloudron every time it crashes seems.... lame.

I think this is the 10th time, just first time I did a good job collecting logs and taking screenshots.

jdaviescoates

@roofboard where are you hosting your Cloudron?

subven

@roofboard do you have the option to restore Cloudron from a backup to a fresh new server? Your server may have been modified (manually installing packages, apt update and so on) before or after installing Cloudron and a fresh reinstall is the easiest and best way to fix this.

roofboard

@subven
Did a few things...

When I came back to the problem today cloudron was down again, this time with too many redirects on all services.
So... I tried upgrading ubuntu, following this guide. https://docs.cloudron.io/guides/upgrade-ubuntu-22/
When I finished the step "apt upgrade" and rebooted the server it started working again. GREAT!!

But... still I moving to a new server sounds like a good idea.
Moving to a new server, 301 redirect issue

So I started with conabo VPS they seem to be a great deal.
I provisioned a 16gb Ram, 200gb machine and installed ubuntu 22
.

After following these instructions to migrate the server from backup
https://docs.cloudron.io/backups/#restore-cloudron

The new server is almost working.

Now, I am having the old problem of too many redirects. All of my services seem to have an infinite loop of 301 redirects pointing to the same location.

history of the too many redirects problem
I have been getting tired of route53 and have an interest in the security features built into cloudflare. So after getting things up an running yesterday I made the switch over to cloudflare. After making that switch I started getting a 301 redirect loop (still on the old server)

After running the apt update / upgrade the problem resolved.

So I figured that I was good to go for moving over to the new VPS.

roofboard

@roofboard Strangely when I set my hosts file to include the line

123.myserver.123.123 domain.com

it starts working and is all happy all of the sudden

NOTE one factor that my confuse matters is that I recently switched to cloudflare so the exact issue which started this thread may not be causing dns issue.

Also... after logging into cloudflare and disabling proxy service it seems to be working.

Question... Is there any issues with Cloudflare proxy?

subven

@roofboard said in Box Crashes.:

After running the apt update / upgrade the problem resolved.

Please be very aware that you are not supposed to apt upgrade on your own (outside the OS upgrade procedure). This may break your installation and might be the reason of trouble you run into.

greherh (2).png

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

Box Crashes.