Can't Restart Nextcloud after Disk full fixed

[Shai]

Hi,

Nextcloud crashed because of disk full. I've cleared out 30G of files. I'm getting the error

Network Error: Network error downloading icon : getaddrinfo EAI_AGAIN api.cloudron.io

when trying to restart the app.

Thanks

[jdaviescoates]

Are all your services (upbound etc) running smoothly?

[Shai]

@jdaviescoates said in Can't Restart Nextcloud after Disk full fixed:

Are all your services (upbound etc) running smoothly?

How do I check that?

All the other apps running on the Cloudron (mostly php apps running Drupal and Wordpress) are running fine now that the disk isn't full.

[girish]

@Shai https://docs.cloudron.io/troubleshooting/#unbound

[Shai]

sudo systemctl status unbound
Unit unbound.service could not be found.

[girish]

@Shai are you running this on web terminal or SSH ? It should be run on SSH. How can unbound disappear?

[Shai]

SSH, unbound seems to be completely gone. nothing at /etc/unbound

[girish]

Might be best to install unbound and ssh then. After that, run /home/yellowtent/box/setup/start.sh

[Shai]

Actually when I said that unbound had disappeared I was wrong. I was logging into proxmox and not into the vm instance that Cloudron is installed on.

So here is what I get when I run sudo systemctl status unbound

unbound.service - Unbound DNS Resolver
     Loaded: loaded (/etc/systemd/system/unbound.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Tue 2023-11-07 19:57:26 UTC; 1 day 18h ago
    Process: 136690 ExecStart=/usr/sbin/unbound -d (code=exited, status=1/FAILURE)
   Main PID: 136690 (code=exited, status=1/FAILURE)

Here is the results when I run sudo journalctl -xe

-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit unbound.service has entered the 'failed' state with result 'exit-code'.
Nov 09 14:48:01 cloudron systemd[1]: Failed to start Unbound DNS Resolver.
-- Subject: A start job for unit unbound.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit unbound.service has finished with a failure.
-- 
-- The job identifier is 6829 and the job result is failed.
Nov 09 14:48:01 cloudron systemd[1]: unbound-resolvconf.service: Start request repeated too quickly.
Nov 09 14:48:01 cloudron systemd[1]: unbound-resolvconf.service: Failed with result 'start-limit-hit'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit unbound-resolvconf.service has entered the 'failed' state with result 'start-limit-hit'.
Nov 09 14:48:01 cloudron systemd[1]: Failed to start Unbound DNS server via resolvconf.
-- Subject: A start job for unit unbound-resolvconf.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit unbound-resolvconf.service has finished with a failure.
-- 
-- The job identifier is 6914 and the job result is failed.
Nov 09 14:48:02 cloudron dockerd[1027]: time="2023-11-09T14:48:02.273332749Z" level=warning msg="[resolver] failed t>
Nov 09 14:48:02 cloudron dockerd[1027]: time="2023-11-09T14:48:02.273332829Z" level=warning msg="[resolver] failed t>
Nov 09 14:48:02 cloudron kernel: Packet dropped: IN=ens18 OUT= MAC=be:b5:b4:54:73:5d:cc:16:7e:fe:d5:bc:08:00 SRC=77.>
Nov 09 14:48:04 cloudron sshd[975466]: Invalid user ubuntu from 157.245.248.106 port 32892
Nov 09 14:48:04 cloudron sshd[975466]: Received disconnect from 157.245.248.106 port 32892:11: Bye Bye [preauth]
Nov 09 14:48:04 cloudron sshd[975466]: Disconnected from invalid user ubuntu 157.245.248.106 port 32892 [preauth]
Nov 09 14:48:07 cloudron containerd[751]: time="2023-11-09T14:48:07.033719922Z" level=info msg="loading plugin \"io.>
Nov 09 14:48:07 cloudron containerd[751]: time="2023-11-09T14:48:07.033820380Z" level=info msg="loading plugin \"io.>
Nov 09 14:48:07 cloudron containerd[751]: time="2023-11-09T14:48:07.033864607Z" level=info msg="loading plugin \"io.>
Nov 09 14:48:07 cloudron containerd[751]: time="2023-11-09T14:48:07.034526543Z" level=info msg="starting signal loop>
Nov 09 14:48:07 cloudron dockerd[1027]: time="2023-11-09T14:48:07.272761873Z" level=warning msg="[resolver] failed t>
Nov 09 14:48:08 cloudron containerd[751]: time="2023-11-09T14:48:08.041835032Z" level=info msg="loading plugin \"io.>
Nov 09 14:48:08 cloudron containerd[751]: time="2023-11-09T14:48:08.042041883Z" level=info msg="loading plugin \"io.>
Nov 09 14:48:08 cloudron containerd[751]: time="2023-11-09T14:48:08.042101487Z" level=info msg="loading plugin \"io.>
Nov 09 14:48:08 cloudron containerd[751]: time="2023-11-09T14:48:08.042903266Z" level=info msg="starting signal loop>
Nov 09 14:48:08 cloudron systemd[135408]: run-docker-runtime\x2drunc-moby-41a3b7fe444b03e8b7a7c6ed9cc3f5a06ce241b926>
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit UNIT has successfully entered the 'dead' state.
Nov 09 14:48:08 cloudron systemd[1]: run-docker-runtime\x2drunc-moby-41a3b7fe444b03e8b7a7c6ed9cc3f5a06ce241b9265d33e>
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit run-docker-runtime\x2drunc-moby-41a3b7fe444b03e8b7a7c6ed9cc3f5a06ce241b9265d33e4aa4fea9710e24e8c-runc.v1>
Nov 09 14:48:08 cloudron dockerd[1027]: time="2023-11-09T14:48:08.272658447Z" level=warning msg="[resolver] failed t>
Nov 09 14:48:08 cloudron dockerd[1027]: time="2023-11-09T14:48:08.292878947Z" level=info msg="ignoring event" contai>
Nov 09 14:48:08 cloudron containerd[751]: time="2023-11-09T14:48:08.294448724Z" level=info msg="shim disconnected" i>
Nov 09 14:48:08 cloudron containerd[751]: time="2023-11-09T14:48:08.294665687Z" level=warning msg="cleaning up after>
Nov 09 14:48:08 cloudron containerd[751]: time="2023-11-09T14:48:08.294743528Z" level=info msg="cleaning up dead shi>
Nov 09 14:48:08 cloudron containerd[751]: time="2023-11-09T14:48:08.306205120Z" level=warning msg="cleanup warnings >
Nov 09 14:48:08 cloudron systemd[1]: var-lib-docker-overlay2-25495d74de46f857703deef2058f832b14629672cc9db85d0768eea>
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit var-lib-docker-overlay2-25495d74de46f857703deef2058f832b14629672cc9db85d0768eea86a9c00a9-merged.mount ha>
Nov 09 14:48:08 cloudron systemd[135408]: var-lib-docker-overlay2-25495d74de46f857703deef2058f832b14629672cc9db85d07>
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit UNIT has successfully entered the 'dead' state.
Nov 09 14:48:09 cloudron dockerd[1027]: time="2023-11-09T14:48:09.218626204Z" level=info msg="ignoring event" contai>
Nov 09 14:48:09 cloudron containerd[751]: time="2023-11-09T14:48:09.218884187Z" level=info msg="shim disconnected" i>
Nov 09 14:48:09 cloudron containerd[751]: time="2023-11-09T14:48:09.218933362Z" level=warning msg="cleaning up after>
Nov 09 14:48:09 cloudron containerd[751]: time="2023-11-09T14:48:09.218944895Z" level=info msg="cleaning up dead shi>
Nov 09 14:48:09 cloudron containerd[751]: time="2023-11-09T14:48:09.229177162Z" level=warning msg="cleanup warnings >
Nov 09 14:48:09 cloudron systemd[135408]: var-lib-docker-overlay2-051dda0fdf9706bfcac8ed411508452ce3341b8a86abc30702>
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit UNIT has successfully entered the 'dead' state.
Nov 09 14:48:09 cloudron systemd[1]: var-lib-docker-overlay2-051dda0fdf9706bfcac8ed411508452ce3341b8a86abc30702a6591>
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit var-lib-docker-overlay2-051dda0fdf9706bfcac8ed411508452ce3341b8a86abc30702a6591e2ccda6ba-merged.mount ha>
Nov 09 14:48:10 cloudron dockerd[1027]: time="2023-11-09T14:48:10.272757872Z" level=warning msg="[resolver] failed t>
Nov 09 14:48:11 cloudron dockerd[1027]: time="2023-11-09T14:48:11.272730404Z" level=warning msg="[resolver] failed t>
Nov 09 14:48:11 cloudron dockerd[1027]: time="2023-11-09T14:48:11.272887082Z" level=warning msg="[resolver] failed t>
Nov 09 14:48:12 cloudron qemu-ga[733]: info: guest-ping called
Nov 09 14:48:13 cloudron dockerd[1027]: time="2023-11-09T14:48:13.272687634Z" level=warning msg="[resolver] failed t>
Nov 09 14:48:13 cloudron dockerd[1027]: time="2023-11-09T14:48:13.551502650Z" level=warning msg="[resolver] failed t>
Nov 09 14:48:14 cloudron dockerd[1027]: time="2023-11-09T14:48:14.068411675Z" level=warning msg="[resolver] failed t>
Nov 09 14:48:14 cloudron dockerd[1027]: time="2023-11-09T14:48:14.272804479Z" level=warning msg="[resolver] failed t>
Nov 09 14:48:17 cloudron dockerd[1027]: time="2023-11-09T14:48:17.272802505Z" level=warning msg="[resolver] failed t>
Nov 09 14:48:18 cloudron dockerd[1027]: time="2023-11-09T14:48:18.272898282Z" level=warning msg="[resolver] failed t>
Nov 09 14:48:19 cloudron dockerd[1027]: time="2023-11-09T14:48:19.272776359Z" level=warning msg="[resolver] failed t>
Nov 09 14:48:19 cloudron dockerd[1027]: time="2023-11-09T14:48:19.272790874Z" level=warning msg="[resolver] failed t>
Nov 09 14:48:21 cloudron sshd[975637]: Invalid user ubuntu from 157.122.157.198 port 36022
Nov 09 14:48:21 cloudron sshd[975637]: Received disconnect from 157.122.157.198 port 36022:11: Bye Bye [preauth]
Nov 09 14:48:21 cloudron sshd[975637]: Disconnected from invalid user ubuntu 157.122.157.198 port 36022 [preauth]
Nov 09 14:48:21 cloudron sudo[975639]: pam_unix(sudo:session): session opened for user root by cece(uid=0)

Thanks in advance for the help on my next step.

[girish]

@Shai ah nice, so did you try https://docs.cloudron.io/troubleshooting/#unbound already?

[Shai]

@girish

unbound-anchor -a /var/lib/unbound/root.key
systemctl restart unbound

That solved it. Thank you.

[Shai]

p.s. @girish Do you think I need to worry that I installed unbound on my Proxmox server when it wasn't already installed. Would it be best to uninstall it? So far that server is behaving normally. (Understood that this question is better for Proxmox support, but since I'm here I thought I'd ask.)

[girish]

@Shai yeah, better to remove it. On ubuntu, installing packages also enables them. So, this might cause problems in the future.