out of space - normal process not working.

[roofboard]

Ran out of space totally embarrassing.
After going through the recovery process.

1. systemctl status unbound, systemctl restart unbound
2. systemctl status nginx, systemctl restart nginx
   • Failed

4448 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE)

3. deleted conf files in /etc/nginx/applications
4. systemctl restart nginx - Successfull.
5. systemctl restart box - successfull.

Now unbound, nginx, and box are running. but the dashboard does not work.

so... I reboot the server.
After rebooting, nginx does not work.

root@my:~# systemctl status nginx
× nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/nginx.service.d
             └─cloudron.conf
     Active: failed (Result: exit-code) since Thu 2022-10-27 17:52:40 UTC; 2min 16s ago
       Docs: man:nginx(8)
    Process: 1194 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE)
        CPU: 19ms

so I deleted the certs
/etc/nginx/cert

and got the same result

I get "Cloudron is offline, Reconnecting..."

---

then i restart box and DNS will not resolve.

Then reboot the whole server. and I am back to this error.

× nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/nginx.service.d
             └─cloudron.conf
     Active: failed (Result: exit-code) since Thu 2022-10-27 18:03:16 UTC; 34s ago
       Docs: man:nginx(8)
    Process: 1181 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE)
        CPU: 10ms

I think I tried running

/home/yellowtent/box/setup/start.sh

and following the unbound troubleshooting step of

forward-zone:
    name: "."
    forward-addr: 1.1.1.1
    forward-addr: 8.8.8.8

into
/etc/unbound/unbound.conf.d/forward-everything.conf and restart unbound:

Any Ideas?

[subven]

Take a look at the logs for nginx with journalctl -u nginx -fa Most of the time you have to delete nginx conf AND cert files and restart box with systemctl restart box.

[roofboard]

@subven said in out of space - normal process not working.:

systemctl restart nginx
Hi, Thanks for the reply.

I just did that, I deleted all files in /etc/nginx/cert/, and /etc/nginx/applications/

Then systemctl restart nginx

And box is running.... everything is running, but cloudron is still down.

when i reboot the server

nginx down and I am back at square 1.

[subven]

@roofboard said in out of space - normal process not working.:

I just did that, I deleted all files in /etc/nginx/cert/, and /etc/nginx/applications/

nginx stuff for Cloudron is at /home/yellowtent/platformdata/nginx/applications Please post your log output from the journalctl command above. Also please restart box with the given command, not nginx.

[roofboard]

@subven
Thanks for the reply. Folowing the instructions.
Here are my commands

root@my:~# cd /home/yellowtent/platformdata/nginx/applications
root@my:/home/yellowtent/platformdata/nginx/applications# ls
default.conf  my.domain.com.conf
root@my:/home/yellowtent/platformdata/nginx/applications# rm *
root@my:/home/yellowtent/platformdata/nginx/applications# systemctl restart box

Below is my log output

root@my:~# journalctl -u nginx -fa
Oct 27 19:04:54 my.domain.com nginx[1200]: nginx: [emerg] cannot load certificate "/home/yellowtent/platformdata/nginx/cert/domain.com.host.cert": BIO_new_file() failed (SSL: error:80000002:system library::No such file or directory:calling fopen(/home/yellowtent/platformdata/nginx/cert/domain.com.host.cert, r) error:10000080:BIO routines::no such file)
Oct 27 19:04:54 my.domain.com nginx[1200]: nginx: configuration file /etc/nginx/nginx.conf test failed
Oct 27 19:04:54 my.domain.com systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Oct 27 19:04:54 my.domain.com systemd[1]: nginx.service: Failed with result 'exit-code'.
Oct 27 19:04:54 my.domain.com systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Oct 27 19:04:54 my.domain.com systemd[1]: nginx.service: Scheduled restart job, restart counter is at 5.
Oct 27 19:04:54 my.domain.com systemd[1]: Stopped A high performance web server and a reverse proxy server.
Oct 27 19:04:54 my.domain.com systemd[1]: nginx.service: Start request repeated too quickly.
Oct 27 19:04:54 my.domain.com systemd[1]: nginx.service: Failed with result 'exit-code'.
Oct 27 19:04:54 my.domain.com systemd[1]: Failed to start A high performance web server and a reverse proxy server.

[subven]

@roofboard please also delete certs under /home/yellowtent/platformdata/nginx/cert/ in addition to the conf at /home/yellowtent/platformdata/nginx/applications/ before restarting box service

Not sure if you are supposed to remove nginx.conf at /home/yellowtent/platformdata/nginx/ so if you touched this file (or the mime.types), this could be an issue. In case you removed nginx.conf and mime.type as well and they are not restored by box, hit me up.

[robi]

Deleting all the certs is a problem I reported recently. There may be a fix, but it's not released yet.

Deleting all the confs is ok as they can regenerate, but not without the one cert for the main cloudron host domain.

See what nginx is looking for? I had to generate that cert manually to get the UI back up.

[roofboard]

@subven
I deleted the files in both the directories, I left the ones in /home/yellowtent/platformdata/nginx alone but inspected them (nothing unusual they are the same as other CL instances. )

then with nginx running I ran systemctl restart box

note - box and nginx were already running.

If i reboot the server nginx will stop running

and all the same my.domain.com never loads always ERR_CONNECTION_RESET

[roofboard]

@robi How do you generate certs manually?

[subven]

@robi hmm I remember I had a similar problem where an stopped app had no cert file which caused nginx to fail at start. I just copied and renamed a random cert to get nginx working at all. The cert for the app I was trying to fix was obviously not right but I was able to run nginx,launch the dashboard, remove the cert from the file system and trigger the renewal process from the UI. Maybe this could work too if you drop something "valid" for the dashboard domain into the nginx cert directory ^^

[roofboard]

@subven @robi
ok. so we are talking about the /home/yellowtent/platformdata/nginx/cert/
cert folder. now i have dummy certs from another CL instance., but which files are needed in that directory? I can get them copied over.

I am guessing maybe... default.cert, and default.key

[subven]

@roofboard [emerg] cannot load certificate "/home/yellowtent/platformdata/nginx/cert/domain.com.host.cert" start with the one that brings your nginx down ^^

[roofboard]

@subven

wordwrap!!! the problem was staring me in the face! But I could not see it because i did not scroll to the right...

Thanks for the help!!!!

I took a cert and key file from another server, and renamed the default cert to be whatever the error wanted.
first thedomain.com.cert, then thedomain.com.key. and so-forth until it loaded. It is working now! .