Help! Server down, classic disk out of space issue

[jdaviescoates]

I immediately rescaled my Hetzner cloud VPS, but that doesn't seem to have taken effect

Oh, it has now!

Last time I run ``df -h it was still showing as 20GB but now showing 80GB, phew (perhaps Hetzner support did something)

But unbound won't restart:

root@ubuntu-2gb-nbg1-1:~# systemctl restart unbound
Job for unbound.service failed because the control process exited with error code.
See "systemctl status unbound.service" and "journalctl -xe" for details.

root@ubuntu-2gb-nbg1-1:~# systemctl status unbound.service
● unbound.service - Unbound DNS Resolver
     Loaded: loaded (/etc/systemd/system/unbound.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Tue 2022-11-08 09:00:18 UTC; 1min 19s ago
    Process: 4078 ExecStart=/usr/sbin/unbound -d (code=exited, status=1/FAILURE)
   Main PID: 4078 (code=exited, status=1/FAILURE)

Nov 08 09:00:18 ubuntu-2gb-nbg1-1 systemd[1]: unbound.service: Scheduled restart job, restart counter is at 5.
Nov 08 09:00:18 ubuntu-2gb-nbg1-1 systemd[1]: Stopped Unbound DNS Resolver.
Nov 08 09:00:18 ubuntu-2gb-nbg1-1 systemd[1]: unbound.service: Start request repeated too quickly.
Nov 08 09:00:18 ubuntu-2gb-nbg1-1 systemd[1]: unbound.service: Failed with result 'exit-code'.
Nov 08 09:00:18 ubuntu-2gb-nbg1-1 systemd[1]: Failed to start Unbound DNS Resolver.
root@ubuntu-2gb-nbg1-1:~# journalctl -xe
-- The job identifier is 3310 and the job result is failed.
Nov 08 09:00:18 ubuntu-2gb-nbg1-1 systemd[1]: unbound-resolvconf.service: Start request repeated too quickly.
Nov 08 09:00:18 ubuntu-2gb-nbg1-1 systemd[1]: unbound-resolvconf.service: Failed with result 'start-limit-hit'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit unbound-resolvconf.service has entered the 'failed' state with result 'start-limit-hit'.
Nov 08 09:00:18 ubuntu-2gb-nbg1-1 systemd[1]: Failed to start Unbound DNS server via resolvconf.
-- Subject: A start job for unit unbound-resolvconf.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit unbound-resolvconf.service has finished with a failure.
-- 
-- The job identifier is 3399 and the job result is failed.
Nov 08 09:00:24 ubuntu-2gb-nbg1-1 sshd[4130]: Invalid user ftpuser from 188.12.166.123 port 40166
Nov 08 09:00:24 ubuntu-2gb-nbg1-1 sshd[4130]: pam_unix(sshd:auth): check pass; user unknown
Nov 08 09:00:24 ubuntu-2gb-nbg1-1 sshd[4130]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=188.12.166.123
Nov 08 09:00:26 ubuntu-2gb-nbg1-1 sshd[4130]: Failed password for invalid user ftpuser from 188.12.166.123 port 40166 ssh2
Nov 08 09:00:27 ubuntu-2gb-nbg1-1 sshd[4130]: Received disconnect from 188.12.166.123 port 40166:11: Bye Bye [preauth]
Nov 08 09:00:27 ubuntu-2gb-nbg1-1 sshd[4130]: Disconnected from invalid user ftpuser 188.12.166.123 port 40166 [preauth]
Nov 08 09:00:42 ubuntu-2gb-nbg1-1 sshd[4152]: Invalid user user from 68.183.20.198 port 53250
Nov 08 09:00:42 ubuntu-2gb-nbg1-1 sshd[4152]: pam_unix(sshd:auth): check pass; user unknown
Nov 08 09:00:42 ubuntu-2gb-nbg1-1 sshd[4152]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=68.183.20.198
Nov 08 09:00:43 ubuntu-2gb-nbg1-1 sshd[4152]: Failed password for invalid user user from 68.183.20.198 port 53250 ssh2
Nov 08 09:00:44 ubuntu-2gb-nbg1-1 sshd[4152]: Received disconnect from 68.183.20.198 port 53250:11: Bye Bye [preauth]
Nov 08 09:00:44 ubuntu-2gb-nbg1-1 sshd[4152]: Disconnected from invalid user user 68.183.20.198 port 53250 [preauth]
Nov 08 09:00:52 ubuntu-2gb-nbg1-1 sshd[4157]: Unable to negotiate with 61.177.173.43 port 29118: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1,diffie-hellman-group14-sha1,diffie>
Nov 08 09:00:55 ubuntu-2gb-nbg1-1 kernel: Packet dropped: IN=eth0 OUT= MAC=96:00:01:a4:bf:6a:d2:74:7f:6e:37:e3:08:00 SRC=180.214.239.113 DST=5.75.159.127 LEN=40 TOS=0x00 PREC=0x00 TTL=240 ID=22387 PROTO=TCP SPT>
Nov 08 09:01:09 ubuntu-2gb-nbg1-1 sshd[4178]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=168.138.7.117  user=root
Nov 08 09:01:12 ubuntu-2gb-nbg1-1 sshd[4178]: Failed password for root from 168.138.7.117 port 40142 ssh2
Nov 08 09:01:12 ubuntu-2gb-nbg1-1 sshd[4178]: Received disconnect from 168.138.7.117 port 40142:11: Bye Bye [preauth]
Nov 08 09:01:12 ubuntu-2gb-nbg1-1 sshd[4178]: Disconnected from authenticating user root 168.138.7.117 port 40142 [preauth]
Nov 08 09:01:15 ubuntu-2gb-nbg1-1 kernel: Packet dropped: IN=eth0 OUT= MAC=96:00:01:a4:bf:6a:d2:74:7f:6e:37:e3:08:00 SRC=91.240.118.196 DST=5.75.159.127 LEN=40 TOS=0x00 PREC=0x00 TTL=245 ID=51602 PROTO=TCP SPT=>
Nov 08 09:01:21 ubuntu-2gb-nbg1-1 sshd[4200]: Invalid user long from 188.12.166.123 port 35550
Nov 08 09:01:21 ubuntu-2gb-nbg1-1 sshd[4200]: pam_unix(sshd:auth): check pass; user unknown
Nov 08 09:01:21 ubuntu-2gb-nbg1-1 sshd[4200]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=188.12.166.123
Nov 08 09:01:23 ubuntu-2gb-nbg1-1 sshd[4200]: Failed password for invalid user long from 188.12.166.123 port 35550 ssh2
Nov 08 09:01:24 ubuntu-2gb-nbg1-1 sshd[4200]: Received disconnect from 188.12.166.123 port 35550:11: Bye Bye [preauth]
Nov 08 09:01:24 ubuntu-2gb-nbg1-1 sshd[4200]: Disconnected from invalid user long 188.12.166.123 port 35550 [preauth]
Nov 08 09:01:32 ubuntu-2gb-nbg1-1 sshd[4204]: Invalid user testuser from 68.183.20.198 port 38718
Nov 08 09:01:32 ubuntu-2gb-nbg1-1 sshd[4204]: pam_unix(sshd:auth): check pass; user unknown
Nov 08 09:01:32 ubuntu-2gb-nbg1-1 sshd[4204]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=68.183.20.198
Nov 08 09:01:34 ubuntu-2gb-nbg1-1 sshd[4204]: Failed password for invalid user testuser from 68.183.20.198 port 38718 ssh2
Nov 08 09:01:35 ubuntu-2gb-nbg1-1 sshd[4204]: Received disconnect from 68.183.20.198 port 38718:11: Bye Bye [preauth]
Nov 08 09:01:35 ubuntu-2gb-nbg1-1 sshd[4204]: Disconnected from invalid user testuser 68.183.20.198 port 38718 [preauth]
Nov 08 09:01:58 ubuntu-2gb-nbg1-1 kernel: Packet dropped: IN=eth0 OUT= MAC=96:00:01:a4:bf:6a:d2:74:7f:6e:37:e3:08:00 SRC=167.248.133.137 DST=5.75.159.127 LEN=44 TOS=0x00 PREC=0x00 TTL=36 ID=55113 PROTO=TCP SPT=>
Nov 08 09:02:18 ubuntu-2gb-nbg1-1 sshd[4265]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=188.12.166.123  user=root
Nov 08 09:02:20 ubuntu-2gb-nbg1-1 sshd[4265]: Failed password for root from 188.12.166.123 port 60370 ssh2
Nov 08 09:02:21 ubuntu-2gb-nbg1-1 sshd[4265]: Received disconnect from 188.12.166.123 port 60370:11: Bye Bye [preauth]
Nov 08 09:02:21 ubuntu-2gb-nbg1-1 sshd[4265]: Disconnected from authenticating user root 188.12.166.123 port 60370 [preauth]

root@ubuntu-2gb-nbg1-1:~# systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/nginx.service.d
             └─cloudron.conf
     Active: active (running) since Tue 2022-11-08 08:57:42 UTC; 9min ago
       Docs: man:nginx(8)
    Process: 663 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
    Process: 2924 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
   Main PID: 2925 (nginx)
      Tasks: 4 (limit: 4502)
     Memory: 12.4M
     CGroup: /system.slice/nginx.service
             ├─2925 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
             ├─3688 nginx: worker process
             ├─3689 nginx: worker process
             └─3690 nginx: worker process

Nov 08 08:57:36 ubuntu-2gb-nbg1-1 systemd[1]: Starting A high performance web server and a reverse proxy server...
Nov 08 08:57:36 ubuntu-2gb-nbg1-1 nginx[663]: nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/safejust.space.>
Nov 08 08:57:41 ubuntu-2gb-nbg1-1 nginx[663]: nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/_.safejust.spac>
Nov 08 08:57:41 ubuntu-2gb-nbg1-1 nginx[663]: nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/_.safejust.spac>
Nov 08 08:57:42 ubuntu-2gb-nbg1-1 nginx[2924]: nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/safejust.space>
Nov 08 08:57:42 ubuntu-2gb-nbg1-1 nginx[2924]: nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/_.safejust.spa>
Nov 08 08:57:42 ubuntu-2gb-nbg1-1 nginx[2924]: nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/_.safejust.spa>
Nov 08 08:57:42 ubuntu-2gb-nbg1-1 systemd[1]: Started A high performance web server and a reverse proxy server.
...skipping...
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/nginx.service.d
             └─cloudron.conf
     Active: active (running) since Tue 2022-11-08 08:57:42 UTC; 9min ago
       Docs: man:nginx(8)
    Process: 663 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
    Process: 2924 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
   Main PID: 2925 (nginx)
      Tasks: 4 (limit: 4502)
     Memory: 12.4M
     CGroup: /system.slice/nginx.service
             ├─2925 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
             ├─3688 nginx: worker process
             ├─3689 nginx: worker process
             └─3690 nginx: worker process

Nov 08 08:57:36 ubuntu-2gb-nbg1-1 systemd[1]: Starting A high performance web server and a reverse proxy server...
Nov 08 08:57:36 ubuntu-2gb-nbg1-1 nginx[663]: nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/safejust.space.>
Nov 08 08:57:41 ubuntu-2gb-nbg1-1 nginx[663]: nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/_.safejust.spac>
Nov 08 08:57:41 ubuntu-2gb-nbg1-1 nginx[663]: nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/_.safejust.spac>
Nov 08 08:57:42 ubuntu-2gb-nbg1-1 nginx[2924]: nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/safejust.space>
Nov 08 08:57:42 ubuntu-2gb-nbg1-1 nginx[2924]: nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/_.safejust.spa>
Nov 08 08:57:42 ubuntu-2gb-nbg1-1 nginx[2924]: nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "r3.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/_.safejust.spa>
Nov 08 08:57:42 ubuntu-2gb-nbg1-1 systemd[1]: Started A high performance web server and a reverse proxy server.

[jdaviescoates]

I think perhaps I've hit the broken certs issues other have had recently... but no idea how to resolve

[girish]

@jdaviescoates https://docs.cloudron.io/troubleshooting/#unbound for unbound. Looks like nginx is up right?

[girish]

@jdaviescoates for the broken certs, just rm /etc/nginx/applications/*. Then, in /etc/nginx/certs just remove all the 0 length files. After that, systemctl restart box. Dashboard should be up after that. Then, go into each app's Location view and click save (this re-generates nginx config of the app).

[jdaviescoates]

@girish thanks, I'd done:

unbound-anchor -a /var/lib/unbound/root.key

previously, but that was before the disk was resized and it didn't seem to help, but just did it again and now I'm back in

Phew!

[jdaviescoates]

But, woah, diskspace fills up quickly when running a Mastodon server!