Docker service failed after docker-compose install
Solved
Support
-
I was installing docker-compose on my Cloudron installation and managed to break everything. Thinking that uninstalling docker-compose would fix it, I gave it a reboot and now I have multiple services failing to start.
I cannot start unbound.
root@cloudron0:~# docker -v
Docker version 20.10.3, build 48d30b5root@cloudron0:/var/lib/docker# systemctl restart docker Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details. root@cloudron0:/var/lib/docker# systemctl status docker.service ● docker.service - LSB: Create lightweight, portable, self-sufficient containers. Loaded: loaded (/etc/init.d/docker; generated) Drop-In: /etc/systemd/system/docker.service.d └─cloudron.conf Active: failed (Result: exit-code) since Fri 2021-08-06 14:44:41 UTC; 9s ago Docs: man:systemd-sysv-generator(8) Process: 19133 ExecStart=/usr/bin/dockerd -H fd:// --log-driver=journald --exec-opt native.cgroupdriver=cgroupfs --sto Aug 06 14:44:41 cloudron0 systemd[1]: Starting LSB: Create lightweight, portable, self-sufficient containers.... Aug 06 14:44:41 cloudron0 systemd[19133]: docker.service: Failed to execute command: No such file or directory Aug 06 14:44:41 cloudron0 systemd[19133]: docker.service: Failed at step EXEC spawning /usr/bin/dockerd: No such file or Aug 06 14:44:41 cloudron0 systemd[1]: docker.service: Control process exited, code=exited status=203 Aug 06 14:44:41 cloudron0 systemd[1]: docker.service: Failed with result 'exit-code'. Aug 06 14:44:41 cloudron0 systemd[1]: Failed to start LSB: Create lightweight, portable, self-sufficient containers..root@cloudron0:/var/lib/docker# systemctl status box ● box.service - Cloudron Admin Loaded: loaded (/etc/systemd/system/box.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2021-08-06 14:43:31 UTC; 745ms ago Main PID: 16120 (node) Tasks: 7 (limit: 4915) CGroup: /system.slice/box.service └─16120 node /home/yellowtent/box/box.js Aug 06 14:43:31 cloudron0 systemd[1]: Started Cloudron Admin. root@cloudron0:/var/lib/docker# systemctl status unbound ● unbound.service - Unbound DNS Resolver Loaded: loaded (/etc/systemd/system/unbound.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Fri 2021-08-06 14:31:38 UTC; 12min ago Process: 23072 ExecStart=/usr/sbin/unbound -d (code=exited, status=1/FAILURE) Main PID: 23072 (code=exited, status=1/FAILURE) Aug 06 14:31:37 cloudron0 systemd[1]: unbound.service: Main process exited, code=exited, status=1/FAILURE Aug 06 14:31:37 cloudron0 systemd[1]: unbound.service: Failed with result 'exit-code'. Aug 06 14:31:38 cloudron0 systemd[1]: unbound.service: Service hold-off time over, scheduling restart. Aug 06 14:31:38 cloudron0 systemd[1]: unbound.service: Scheduled restart job, restart counter is at 5. Aug 06 14:31:38 cloudron0 systemd[1]: Stopped Unbound DNS Resolver. Aug 06 14:31:38 cloudron0 systemd[1]: unbound.service: Start request repeated too quickly. Aug 06 14:31:38 cloudron0 systemd[1]: unbound.service: Failed with result 'exit-code'. Aug 06 14:31:38 cloudron0 systemd[1]: Failed to start Unbound DNS Resolver. root@cloudron0:/var/lib/docker# unbound-anchor -a /var/lib/unbound/root.key root@cloudron0:/var/lib/docker# systemctl restart unbound root@cloudron0:/var/lib/docker# systemctl status unbound ● unbound.service - Unbound DNS Resolver Loaded: loaded (/etc/systemd/system/unbound.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Fri 2021-08-06 14:43:53 UTC; 2s ago Process: 17316 ExecStart=/usr/sbin/unbound -d (code=exited, status=1/FAILURE) Main PID: 17316 (code=exited, status=1/FAILURE) Aug 06 14:43:52 cloudron0 systemd[1]: unbound.service: Main process exited, code=exited, status=1/FAILURE Aug 06 14:43:52 cloudron0 systemd[1]: unbound.service: Failed with result 'exit-code'. Aug 06 14:43:53 cloudron0 systemd[1]: unbound.service: Service hold-off time over, scheduling restart. Aug 06 14:43:53 cloudron0 systemd[1]: unbound.service: Scheduled restart job, restart counter is at 5. Aug 06 14:43:53 cloudron0 systemd[1]: Stopped Unbound DNS Resolver. Aug 06 14:43:53 cloudron0 systemd[1]: unbound.service: Start request repeated too quickly. Aug 06 14:43:53 cloudron0 systemd[1]: unbound.service: Failed with result 'exit-code'. Aug 06 14:43:53 cloudron0 systemd[1]: Failed to start Unbound DNS Resolver.MySQL and Nginx are running, and the files in
/home/yellowtent/appsdataare intactAny help would be appreciated.
-
@zezaung Looks like maybe uninstalling compose also nuked docker from what I can tell. The logs say
docker.service: Failed at step EXEC spawning /usr/bin/dockerd. I guess that means the docker binary disappeared. Can you check if/var/lib/dockeris still there? If that too is gone, we have to re-create all the containers (I can explain how to do this later).We install docker from the upstream repos directory, so download the 3 debs below and install them using
apt install -y. docker_version below is 20.10.3 and ubuntu_codename is the output oflsb_release -cshttps://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/containerd.io_1.4.3-1_amd64.deb https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce-cli_${docker_version}~3-0~ubuntu-${ubuntu_codename}_amd64.deb https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce_${docker_version}~3-0~ubuntu-${ubuntu_codename}_amd64.deb -
If it helps, the relevant code we use is here - https://git.cloudron.io/cloudron/box/-/blob/master/scripts/installer.sh#L77 (but you cannot run the installer script directly since it's meant to be called from a new box update).
-
@girish I can't thank you enough. Had some difficulties with the
-o /tmp/filename.deb(it wasn't saving the file) so I downloaded and renamed the .deb onto my computer, filezilla'd to my home directory, then usedmv filename.deb /tmpI restarted unbound and everything started running perfectly
Here is the result:
The lib files seemed untouchedroot@cloudron0:/var/lib/docker# ls -al total 200 drwx--x--x 14 root root 4096 Aug 6 14:31 . drwxr-xr-x 47 root root 4096 Dec 10 2020 .. drwx------ 2 root root 4096 Dec 10 2020 builder drwx--x--x 4 root root 4096 Mar 15 03:35 buildkit drwx-----x 59 root root 12288 Aug 6 13:32 containers drwx------ 3 root root 4096 Dec 10 2020 image drwxr-x--- 3 root root 4096 Dec 10 2020 network drwx-----x 507 root root 73728 Aug 6 13:57 overlay2 drwx------ 4 root root 4096 Dec 10 2020 plugins drwx------ 2 root root 4096 Aug 6 13:57 runtimes drwx------ 2 root root 4096 Dec 10 2020 swarm drwx------ 2 root root 4096 Aug 6 13:57 tmp drwx------ 2 root root 4096 Dec 10 2020 trust drwx-----x 458 root root 61440 Aug 6 13:57 volumes -rw-r--r-- 1 root root 1394 Aug 6 14:31 'ystemctl status unbound'I hope this worked the way it should:
root@cloudron0:/tmp# apt install -y /tmp/containerd.deb Reading package lists... Done Building dependency tree Reading state information... Done Note, selecting 'containerd.io' instead of '/tmp/containerd.deb' The following packages were automatically installed and are no longer required: amd64-microcode aufs-tools cgroupfs-mount golang-docker-credential-helpers intel-microcode iucode-tool libsecret-1-0 libsecret-common pigz python-asn1crypto python-backports.ssl-match-hostname python-cached-property python-certifi python-cffi-backend python-chardet python-cryptography python-docker python-dockerpty python-dockerpycreds python-docopt python-enum34 python-funcsigs python-functools32 python-idna python-ipaddress python-jsonschema python-mock python-openssl python-pbr python-pkg-resources python-requests python-six python-texttable python-urllib3 python-websocket python-yaml Use 'apt autoremove' to remove them. The following NEW packages will be installed: containerd.io 0 upgraded, 1 newly installed, 0 to remove and 43 not upgraded. After this operation, 129 MB of additional disk space will be used. Get:1 /tmp/containerd.deb containerd.io amd64 1.4.3-1 [28.1 MB] Selecting previously unselected package containerd.io. (Reading database ... 151160 files and directories currently installed.) Preparing to unpack /tmp/containerd.deb ... Reading package lists... Done Building dependency tree Reading state information... Done Note, selecting 'docker-ce-cli' instead of '/tmp/docker-ce-cli.deb' docker-ce-cli is already the newest version (5:20.10.3~3-0~ubuntu-bionic). The following packages were automatically installed and are no longer required: amd64-microcode aufs-tools cgroupfs-mount golang-docker-credential-helpers intel-microcode iucode-tool libsecret-1-0 libsecret-common pigz python-asn1crypto python-backports.ssl-match-hostname python-cached-property python-certifi python-cffi-backend python-chardet python-cryptography python-docker python-dockerpty python-dockerpycreds python-docopt python-enum34 python-funcsigs python-functools32 python-idna python-ipaddress python-jsonschema python-mock python-openssl python-pbr python-pkg-resources python-requests python-six python-texttable python-urllib3 python-websocket python-yaml Use 'apt autoremove' to remove them. 0 upgraded, 0 newly installed, 0 to remove and 43 not upgraded. root@cloudron0:/tmp# apt install -y /tmp/docker.deb Reading package lists... Done Building dependency tree Reading state information... Done Note, selecting 'docker-ce' instead of '/tmp/docker.deb' The following packages were automatically installed and are no longer required: amd64-microcode golang-docker-credential-helpers intel-microcode iucode-tool libsecret-1-0 libsecret-common python-asn1crypto python-backports.ssl-match-hostname python-cached-property python-certifi python-cffi-backend python-chardet python-cryptography python-docker python-dockerpty python-dockerpycreds python-docopt python-enum34 python-funcsigs python-functools32 python-idna python-ipaddress python-jsonschema python-mock python-openssl python-pbr python-pkg-resources python-requests python-six python-texttable python-urllib3 python-websocket python-yaml Use 'apt autoremove' to remove them. Recommended packages: docker-ce-rootless-extras The following NEW packages will be installed: docker-ce 0 upgraded, 1 newly installed, 0 to remove and 43 not upgraded. After this operation, 121 MB of additional disk space will be used. Get:1 /tmp/docker.deb docker-ce amd64 5:20.10.3~3-0~ubuntu-bionic [24.8 MB] Selecting previously unselected package docker-ce. (Reading database ... 151174 files and directories currently installed.) Preparing to unpack /tmp/docker.deb ... Unpacking docker-ce (5:20.10.3~3-0~ubuntu-bionic) ... Setting up docker-ce (5:20.10.3~3-0~ubuntu-bionic) ... Processing triggers for systemd (237-3ubuntu10.43) ...#############################.................................] Processing triggers for ureadahead (0.100.0-21) ...root@cloudron0:/tmp# systemctl status box ● box.service - Cloudron Admin Loaded: loaded (/etc/systemd/system/box.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2021-08-06 18:49:09 UTC; 3min 8s ago Main PID: 21658 (node) Tasks: 11 (limit: 4915) CGroup: /system.slice/box.service └─21658 node /home/yellowtent/box/box.js Aug 06 18:49:09 cloudron0 systemd[1]: Started Cloudron Admin. Aug 06 18:49:16 cloudron0 sudo[24775]: pam_unix(sudo:session): session opened for user root by (uid=0) Aug 06 18:49:16 cloudron0 sudo[24775]: pam_unix(sudo:session): session closed for user root Aug 06 18:49:16 cloudron0 sudo[24838]: pam_unix(sudo:session): session opened for user root by (uid=0) Aug 06 18:49:16 cloudron0 sudo[24838]: pam_unix(sudo:session): session closed for user root Aug 06 18:49:47 cloudron0 sudo[29454]: pam_unix(sudo:session): session opened for user root by (uid=0) Aug 06 18:49:47 cloudron0 sudo[29454]: pam_unix(sudo:session): session closed for user root root@cloudron0:/tmp# systemctl status unbound ● unbound.service - Unbound DNS Resolver Loaded: loaded (/etc/systemd/system/unbound.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Fri 2021-08-06 14:45:47 UTC; 4h 7min ago Main PID: 1346 (code=exited, status=1/FAILURE) Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable. root@cloudron0:/tmp# unbound-anchor -a /var/lib/unbound/root.key boundroot@cloudron0:/tmp# systemctl restart unbound root@cloudron0:/tmp# systemctl status unbound ● unbound.service - Unbound DNS Resolver Loaded: loaded (/etc/systemd/system/unbound.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2021-08-06 18:53:01 UTC; 3s ago Main PID: 4506 (unbound) Tasks: 1 (limit: 4915) CGroup: /system.slice/unbound.service └─4506 /usr/sbin/unbound -d Aug 06 18:53:01 cloudron0 systemd[1]: Started Unbound DNS Resolver. Aug 06 18:53:01 cloudron0 unbound[4506]: [4506:0] notice: init module 0: subnet Aug 06 18:53:01 cloudron0 unbound[4506]: [4506:0] notice: init module 1: validator Aug 06 18:53:01 cloudron0 unbound[4506]: [4506:0] notice: init module 2: iterator Aug 06 18:53:01 cloudron0 unbound[4506]: [4506:0] info: start of service (unbound 1.6.7). root@cloudron0:/tmp# systemctl status nginx ● nginx.service - nginx - high performance web server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/nginx.service.d └─cloudron.conf Active: active (running) since Fri 2021-08-06 14:45:45 UTC; 4h 7min ago Docs: http://nginx.org/en/docs/ Main PID: 980 (nginx) Tasks: 5 (limit: 4915) CGroup: /system.slice/nginx.service ├─ 980 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf ├─29499 nginx: worker process ├─29500 nginx: worker process ├─29501 nginx: worker process └─29502 nginx: worker process Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable. root@cloudron0:/tmp# systemctl status mysql ● mysql.service - MySQL Community Server Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2021-08-06 14:45:44 UTC; 4h 7min ago Main PID: 756 (mysqld) Tasks: 32 (limit: 4915) CGroup: /system.slice/mysql.service └─756 /usr/sbin/mysqld --daemonize --pid-file=/run/mysqld/mysqld.pid Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable. -
Today I also managed to completely destroy Cloudron by installing Docker-Compose (apt install docker-compose) today. I think you should prevent exactly that during the installation.
-
@johannesjom generally it is not supported to install other things via apt manually. There are some warnings about this when you SSH into your server, but we can't quite prevent users who have root access to their own server to install things or also change essential config files. So not sure what we can do here besides warning the user.
