Nginx problems running after diskfull recovery walkthrough
-
Hello there
I am a big fan of cloudron. One of the other users of our cloudron instance filled up the drive with too much 4k drone footage. As expected this caused some issues for cloudron.
However, when attempting to follow the instructions laid out for recovery in this situation, we've struggled to make sense of how to get nginx restarted.
After poking around on this forum I found this page which seems to have mirrored what we were up against: https://forum.cloudron.io/topic/7219/cannot-load-certificate-key-_-sub-domain-org-key-failed-no-such-file-or-directory
^ We did however have a key but it was blank and nginx was complaining that the data found within it was wrong.
So, we ran mv on the associated cert & keyfiles & conf, and attempted a restart, however that doesn't work and the trail has run dry with these cryptic nginx journal logs which do not suggest what we should do next to debug.
-- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Unit box.service has finished shutting down. Sep 25 14:24:09 cloudron0 systemd[1]: Started Cloudron Admin. -- Subject: Unit box.service has finished start-up -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Unit box.service has finished starting up. -- -- The start-up result is RESULT. Sep 25 14:24:10 cloudron0 systemd[1]: box.service: Main process exited, code=exited, status=1/FAILURE Sep 25 14:24:10 cloudron0 systemd[1]: box.service: Failed with result 'exit-code'. Sep 25 14:24:10 cloudron0 systemd[1]: box.service: Service hold-off time over, scheduling restart. Sep 25 14:24:10 cloudron0 systemd[1]: box.service: Scheduled restart job, restart counter is at 159233. -- Subject: Automatic restarting of a unit has been scheduled -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Automatic restarting of the unit box.service has been scheduled, as the result for -- the configured Restart= setting for the unit. Sep 25 14:24:10 cloudron0 systemd[1]: Stopped Cloudron Admin. -- Subject: Unit box.service has finished shutting down -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Unit box.service has finished shutting down. Sep 25 14:24:10 cloudron0 systemd[1]: Started Cloudron Admin. -- Subject: Unit box.service has finished start-up -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Unit box.service has finished starting up. -- -- The start-up result is RESULT. Sep 25 14:24:11 cloudron0 systemd[1]: box.service: Main process exited, code=exited, status=1/FAILURE Sep 25 14:24:11 cloudron0 systemd[1]: box.service: Failed with result 'exit-code'. Sep 25 14:24:11 cloudron0 systemd[1]: box.service: Service hold-off time over, scheduling restart. Sep 25 14:24:11 cloudron0 systemd[1]: box.service: Scheduled restart job, restart counter is at 159234. -- Subject: Automatic restarting of a unit has been scheduled -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Automatic restarting of the unit box.service has been scheduled, as the result for -- the configured Restart= setting for the unit. Sep 25 14:24:11 cloudron0 systemd[1]: Stopped Cloudron Admin. -- Subject: Unit box.service has finished shutting down -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Unit box.service has finished shutting down. Sep 25 14:24:11 cloudron0 systemd[1]: Started Cloudron Admin. -- Subject: Unit box.service has finished start-up -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Unit box.service has finished starting up. -- -- The start-up result is RESULT. Sep 25 14:24:12 cloudron0 systemd[1]: box.service: Main process exited, code=exited, status=1/FAILURE Sep 25 14:24:12 cloudron0 systemd[1]: box.service: Failed with result 'exit-code'. Sep 25 14:24:12 cloudron0 systemd[1]: box.service: Service hold-off time over, scheduling restart. Sep 25 14:24:12 cloudron0 systemd[1]: box.service: Scheduled restart job, restart counter is at 159235. -- Subject: Automatic restarting of a unit has been scheduled -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Automatic restarting of the unit box.service has been scheduled, as the result for -- the configured Restart= setting for the unit. Sep 25 14:24:12 cloudron0 systemd[1]: Stopped Cloudron Admin. -- Subject: Unit box.service has finished shutting down -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Unit box.service has finished shutting down. Sep 25 14:24:12 cloudron0 systemd[1]: Started Cloudron Admin. -- Subject: Unit box.service has finished start-up``` -
by the way, when I run restart on nginx I get the very same issue experienced by the other user i linked to in that thread
Here is the newest log (note the "BIO")
Sep 25 14:27:28 cloudron0 systemd[1]: Starting nginx - high performance web server... -- Subject: Unit nginx.service has begun start-up -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Unit nginx.service has begun starting up. Sep 25 14:27:28 cloudron0 nginx[20372]: nginx: [emerg] cannot load certificate "/home/yellowtent/platformdata/nginx/cert/_.covidian.life.cert": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/home/yellowtent/platformdata/nginx/cert/_.covidian.life.cert','r') error:2006D080:BIO routines:BIO_new_file:no such file) Sep 25 14:27:28 cloudron0 systemd[1]: nginx.service: Control process exited, code=exited status=1 Sep 25 14:27:28 cloudron0 systemd[1]: nginx.service: Failed with result 'exit-code'. Sep 25 14:27:28 cloudron0 systemd[1]: Failed to start nginx - high performance web server. -- Subject: Unit nginx.service has failed -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Unit nginx.service has failed. -- -- The result is RESULT. Sep 25 14:27:28 cloudron0 systemd[1]: Started Cloudron Admin. -- Subject: Unit box.service has finished start-up -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Unit box.service has finished starting up. -- -- The start-up result is RESULT. Sep 25 14:27:28 cloudron0 systemd[1]: nginx.service: Service hold-off time over, scheduling restart. Sep 25 14:27:28 cloudron0 systemd[1]: nginx.service: Scheduled restart job, restart counter is at 5. -- Subject: Automatic restarting of a unit has been scheduled -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Automatic restarting of the unit nginx.service has been scheduled, as the result for -- the configured Restart= setting for the unit. Sep 25 14:27:28 cloudron0 systemd[1]: Stopped nginx - high performance web server. -- Subject: Unit nginx.service has finished shutting down -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Unit nginx.service has finished shutting down. Sep 25 14:27:28 cloudron0 systemd[1]: nginx.service: Start request repeated too quickly. Sep 25 14:27:28 cloudron0 systemd[1]: nginx.service: Failed with result 'exit-code'. Sep 25 14:27:28 cloudron0 systemd[1]: Failed to start nginx - high performance web server. -- Subject: Unit nginx.service has failed -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Unit nginx.service has failed. -- -- The result is RESULT. Sep 25 14:27:29 cloudron0 systemd[1]: box.service: Main process exited, code=exited, status=1/FAILURE Sep 25 14:27:29 cloudron0 systemd[1]: box.service: Failed with result 'exit-code'. Sep 25 14:27:29 cloudron0 systemd[1]: box.service: Service hold-off time over, scheduling restart. Sep 25 14:27:29 cloudron0 systemd[1]: box.service: Scheduled restart job, restart counter is at 159431. -- Subject: Automatic restarting of a unit has been scheduled -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Automatic restarting of the unit box.service has been scheduled, as the result for -- the configured Restart= setting for the unit. Sep 25 14:27:29 cloudron0 systemd[1]: Stopped Cloudron Admin. -- Subject: Unit box.service has finished shutting down -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Unit box.service has finished shutting down. Sep 25 14:27:29 cloudron0 systemd[1]: Started Cloudron Admin. -- Subject: Unit box.service has finished start-up -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Unit box.service has finished starting up. -- -- The start-up result is RESULT. -
So turns out we've resolved this portion.
Main issue is there were way too many confs pointing to the main problematic certficate. Everything in /home/yellowtent/platformdata/nginx/applications basically needed to be moved (/removed) in order to get nginx to restart again once more
-
Z ZeZaung referenced this topic on