Help! Server down, classic disk out of space issue
-
I immediately rescaled my Hetzner cloud VPS, but that doesn't seem to have taken effect
Oh, it has now!
Last time I run ``df -h it was still showing as 20GB but now showing 80GB, phew (perhaps Hetzner support did something)
But unbound won't restart:
root@ubuntu-2gb-nbg1-1:~# systemctl restart unbound Job for unbound.service failed because the control process exited with error code. See "systemctl status unbound.service" and "journalctl -xe" for details.
root@ubuntu-2gb-nbg1-1:~# systemctl status unbound.service ● unbound.service - Unbound DNS Resolver Loaded: loaded (/etc/systemd/system/unbound.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2022-11-08 09:00:18 UTC; 1min 19s ago Process: 4078 ExecStart=/usr/sbin/unbound -d (code=exited, status=1/FAILURE) Main PID: 4078 (code=exited, status=1/FAILURE) Nov 08 09:00:18 ubuntu-2gb-nbg1-1 systemd[1]: unbound.service: Scheduled restart job, restart counter is at 5. Nov 08 09:00:18 ubuntu-2gb-nbg1-1 systemd[1]: Stopped Unbound DNS Resolver. Nov 08 09:00:18 ubuntu-2gb-nbg1-1 systemd[1]: unbound.service: Start request repeated too quickly. Nov 08 09:00:18 ubuntu-2gb-nbg1-1 systemd[1]: unbound.service: Failed with result 'exit-code'. Nov 08 09:00:18 ubuntu-2gb-nbg1-1 systemd[1]: Failed to start Unbound DNS Resolver. root@ubuntu-2gb-nbg1-1:~# journalctl -xe -- The job identifier is 3310 and the job result is failed. Nov 08 09:00:18 ubuntu-2gb-nbg1-1 systemd[1]: unbound-resolvconf.service: Start request repeated too quickly. Nov 08 09:00:18 ubuntu-2gb-nbg1-1 systemd[1]: unbound-resolvconf.service: Failed with result 'start-limit-hit'. -- Subject: Unit failed -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- The unit unbound-resolvconf.service has entered the 'failed' state with result 'start-limit-hit'. Nov 08 09:00:18 ubuntu-2gb-nbg1-1 systemd[1]: Failed to start Unbound DNS server via resolvconf. -- Subject: A start job for unit unbound-resolvconf.service has failed -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- A start job for unit unbound-resolvconf.service has finished with a failure. -- -- The job identifier is 3399 and the job result is failed. Nov 08 09:00:24 ubuntu-2gb-nbg1-1 sshd[4130]: Invalid user ftpuser from 188.12.166.123 port 40166 Nov 08 09:00:24 ubuntu-2gb-nbg1-1 sshd[4130]: pam_unix(sshd:auth): check pass; user unknown Nov 08 09:00:24 ubuntu-2gb-nbg1-1 sshd[4130]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=188.12.166.123 Nov 08 09:00:26 ubuntu-2gb-nbg1-1 sshd[4130]: Failed password for invalid user ftpuser from 188.12.166.123 port 40166 ssh2 Nov 08 09:00:27 ubuntu-2gb-nbg1-1 sshd[4130]: Received disconnect from 188.12.166.123 port 40166:11: Bye Bye [preauth] Nov 08 09:00:27 ubuntu-2gb-nbg1-1 sshd[4130]: Disconnected from invalid user ftpuser 188.12.166.123 port 40166 [preauth] Nov 08 09:00:42 ubuntu-2gb-nbg1-1 sshd[4152]: Invalid user user from 68.183.20.198 port 53250 Nov 08 09:00:42 ubuntu-2gb-nbg1-1 sshd[4152]: pam_unix(sshd:auth): check pass; user unknown Nov 08 09:00:42 ubuntu-2gb-nbg1-1 sshd[4152]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=68.183.20.198 Nov 08 09:00:43 ubuntu-2gb-nbg1-1 sshd[4152]: Failed password for invalid user user from 68.183.20.198 port 53250 ssh2 Nov 08 09:00:44 ubuntu-2gb-nbg1-1 sshd[4152]: Received disconnect from 68.183.20.198 port 53250:11: Bye Bye [preauth] Nov 08 09:00:44 ubuntu-2gb-nbg1-1 sshd[4152]: Disconnected from invalid user user 68.183.20.198 port 53250 [preauth] Nov 08 09:00:52 ubuntu-2gb-nbg1-1 sshd[4157]: Unable to negotiate with 61.177.173.43 port 29118: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1,diffie-hellman-group14-sha1,diffie> Nov 08 09:00:55 ubuntu-2gb-nbg1-1 kernel: Packet dropped: IN=eth0 OUT= MAC=96:00:01:a4:bf:6a:d2:74:7f:6e:37:e3:08:00 SRC=180.214.239.113 DST=5.75.159.127 LEN=40 TOS=0x00 PREC=0x00 TTL=240 ID=22387 PROTO=TCP SPT> Nov 08 09:01:09 ubuntu-2gb-nbg1-1 sshd[4178]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=168.138.7.117 user=root Nov 08 09:01:12 ubuntu-2gb-nbg1-1 sshd[4178]: Failed password for root from 168.138.7.117 port 40142 ssh2 Nov 08 09:01:12 ubuntu-2gb-nbg1-1 sshd[4178]: Received disconnect from 168.138.7.117 port 40142:11: Bye Bye [preauth] Nov 08 09:01:12 ubuntu-2gb-nbg1-1 sshd[4178]: Disconnected from authenticating user root 168.138.7.117 port 40142 [preauth] Nov 08 09:01:15 ubuntu-2gb-nbg1-1 kernel: Packet dropped: IN=eth0 OUT= MAC=96:00:01:a4:bf:6a:d2:74:7f:6e:37:e3:08:00 SRC=91.240.118.196 DST=5.75.159.127 LEN=40 TOS=0x00 PREC=0x00 TTL=245 ID=51602 PROTO=TCP SPT=> Nov 08 09:01:21 ubuntu-2gb-nbg1-1 sshd[4200]: Invalid user long from 188.12.166.123 port 35550 Nov 08 09:01:21 ubuntu-2gb-nbg1-1 sshd[4200]: pam_unix(sshd:auth): check pass; user unknown Nov 08 09:01:21 ubuntu-2gb-nbg1-1 sshd[4200]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=188.12.166.123 Nov 08 09:01:23 ubuntu-2gb-nbg1-1 sshd[4200]: Failed password for invalid user long from 188.12.166.123 port 35550 ssh2 Nov 08 09:01:24 ubuntu-2gb-nbg1-1 sshd[4200]: Received disconnect from 188.12.166.123 port 35550:11: Bye Bye [preauth] Nov 08 09:01:24 ubuntu-2gb-nbg1-1 sshd[4200]: Disconnected from invalid user long 188.12.166.123 port 35550 [preauth] Nov 08 09:01:32 ubuntu-2gb-nbg1-1 sshd[4204]: Invalid user testuser from 68.183.20.198 port 38718 Nov 08 09:01:32 ubuntu-2gb-nbg1-1 sshd[4204]: pam_unix(sshd:auth): check pass; user unknown Nov 08 09:01:32 ubuntu-2gb-nbg1-1 sshd[4204]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=68.183.20.198 Nov 08 09:01:34 ubuntu-2gb-nbg1-1 sshd[4204]: Failed password for invalid user testuser from 68.183.20.198 port 38718 ssh2 Nov 08 09:01:35 ubuntu-2gb-nbg1-1 sshd[4204]: Received disconnect from 68.183.20.198 port 38718:11: Bye Bye [preauth] Nov 08 09:01:35 ubuntu-2gb-nbg1-1 sshd[4204]: Disconnected from invalid user testuser 68.183.20.198 port 38718 [preauth] Nov 08 09:01:58 ubuntu-2gb-nbg1-1 kernel: Packet dropped: IN=eth0 OUT= MAC=96:00:01:a4:bf:6a:d2:74:7f:6e:37:e3:08:00 SRC=167.248.133.137 DST=5.75.159.127 LEN=44 TOS=0x00 PREC=0x00 TTL=36 ID=55113 PROTO=TCP SPT=> Nov 08 09:02:18 ubuntu-2gb-nbg1-1 sshd[4265]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=188.12.166.123 user=root Nov 08 09:02:20 ubuntu-2gb-nbg1-1 sshd[4265]: Failed password for root from 188.12.166.123 port 60370 ssh2 Nov 08 09:02:21 ubuntu-2gb-nbg1-1 sshd[4265]: Received disconnect from 188.12.166.123 port 60370:11: Bye Bye [preauth] Nov 08 09:02:21 ubuntu-2gb-nbg1-1 sshd[4265]: Disconnected from authenticating user root 188.12.166.123 port 60370 [preauth]
root@ubuntu-2gb-nbg1-1:~# systemctl status nginx ● nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/nginx.service.d └─cloudron.conf Active: active (running) since Tue 2022-11-08 08:57:42 UTC; 9min ago Docs: man:nginx(8) Process: 663 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Process: 2924 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Main PID: 2925 (nginx) Tasks: 4 (limit: 4502) Memory: 12.4M CGroup: /system.slice/nginx.service ├─2925 nginx: master process /usr/sbin/nginx -g daemon on; master_process on; ├─3688 nginx: worker process ├─3689 nginx: worker process └─3690 nginx: worker process Nov 08 08:57:36 ubuntu-2gb-nbg1-1 systemd[1]: Starting A high performance web server and a reverse proxy server... Nov 08 08:57:36 ubuntu-2gb-nbg1-1 nginx[663]: nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/safejust.space.> Nov 08 08:57:41 ubuntu-2gb-nbg1-1 nginx[663]: nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/_.safejust.spac> Nov 08 08:57:41 ubuntu-2gb-nbg1-1 nginx[663]: nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/_.safejust.spac> Nov 08 08:57:42 ubuntu-2gb-nbg1-1 nginx[2924]: nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/safejust.space> Nov 08 08:57:42 ubuntu-2gb-nbg1-1 nginx[2924]: nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/_.safejust.spa> Nov 08 08:57:42 ubuntu-2gb-nbg1-1 nginx[2924]: nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/_.safejust.spa> Nov 08 08:57:42 ubuntu-2gb-nbg1-1 systemd[1]: Started A high performance web server and a reverse proxy server. ...skipping... ● nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/nginx.service.d └─cloudron.conf Active: active (running) since Tue 2022-11-08 08:57:42 UTC; 9min ago Docs: man:nginx(8) Process: 663 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Process: 2924 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Main PID: 2925 (nginx) Tasks: 4 (limit: 4502) Memory: 12.4M CGroup: /system.slice/nginx.service ├─2925 nginx: master process /usr/sbin/nginx -g daemon on; master_process on; ├─3688 nginx: worker process ├─3689 nginx: worker process └─3690 nginx: worker process Nov 08 08:57:36 ubuntu-2gb-nbg1-1 systemd[1]: Starting A high performance web server and a reverse proxy server... Nov 08 08:57:36 ubuntu-2gb-nbg1-1 nginx[663]: nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/safejust.space.> Nov 08 08:57:41 ubuntu-2gb-nbg1-1 nginx[663]: nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/_.safejust.spac> Nov 08 08:57:41 ubuntu-2gb-nbg1-1 nginx[663]: nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/_.safejust.spac> Nov 08 08:57:42 ubuntu-2gb-nbg1-1 nginx[2924]: nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/safejust.space> Nov 08 08:57:42 ubuntu-2gb-nbg1-1 nginx[2924]: nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/_.safejust.spa> Nov 08 08:57:42 ubuntu-2gb-nbg1-1 nginx[2924]: nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/_.safejust.spa> Nov 08 08:57:42 ubuntu-2gb-nbg1-1 systemd[1]: Started A high performance web server and a reverse proxy server.
-
I think perhaps I've hit the broken certs issues other have had recently... but no idea how to resolve
-
@jdaviescoates https://docs.cloudron.io/troubleshooting/#unbound for unbound. Looks like nginx is up right?
-
@jdaviescoates for the broken certs, just
rm /etc/nginx/applications/*
. Then, in/etc/nginx/certs
just remove all the 0 length files. After that,systemctl restart box
. Dashboard should be up after that. Then, go into each app's Location view and click save (this re-generates nginx config of the app). -
-
-
But, woah, diskspace fills up quickly when running a Mastodon server!