Docker service failed after docker-compose install

[ZeZaung]

I was installing docker-compose on my Cloudron installation and managed to break everything. Thinking that uninstalling docker-compose would fix it, I gave it a reboot and now I have multiple services failing to start.

I cannot start unbound.

root@cloudron0:~# docker -v
Docker version 20.10.3, build 48d30b5

root@cloudron0:/var/lib/docker# systemctl restart docker
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.
root@cloudron0:/var/lib/docker# systemctl status docker.service
● docker.service - LSB: Create lightweight, portable, self-sufficient containers.
   Loaded: loaded (/etc/init.d/docker; generated)
  Drop-In: /etc/systemd/system/docker.service.d
           └─cloudron.conf
   Active: failed (Result: exit-code) since Fri 2021-08-06 14:44:41 UTC; 9s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 19133 ExecStart=/usr/bin/dockerd -H fd:// --log-driver=journald --exec-opt native.cgroupdriver=cgroupfs --sto

Aug 06 14:44:41 cloudron0 systemd[1]: Starting LSB: Create lightweight, portable, self-sufficient containers....
Aug 06 14:44:41 cloudron0 systemd[19133]: docker.service: Failed to execute command: No such file or directory
Aug 06 14:44:41 cloudron0 systemd[19133]: docker.service: Failed at step EXEC spawning /usr/bin/dockerd: No such file or
Aug 06 14:44:41 cloudron0 systemd[1]: docker.service: Control process exited, code=exited status=203
Aug 06 14:44:41 cloudron0 systemd[1]: docker.service: Failed with result 'exit-code'.
Aug 06 14:44:41 cloudron0 systemd[1]: Failed to start LSB: Create lightweight, portable, self-sufficient containers..

root@cloudron0:/var/lib/docker# systemctl status box
● box.service - Cloudron Admin
   Loaded: loaded (/etc/systemd/system/box.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2021-08-06 14:43:31 UTC; 745ms ago
 Main PID: 16120 (node)
    Tasks: 7 (limit: 4915)
   CGroup: /system.slice/box.service
           └─16120 node /home/yellowtent/box/box.js

Aug 06 14:43:31 cloudron0 systemd[1]: Started Cloudron Admin.
root@cloudron0:/var/lib/docker# systemctl status unbound
● unbound.service - Unbound DNS Resolver
   Loaded: loaded (/etc/systemd/system/unbound.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2021-08-06 14:31:38 UTC; 12min ago
  Process: 23072 ExecStart=/usr/sbin/unbound -d (code=exited, status=1/FAILURE)
 Main PID: 23072 (code=exited, status=1/FAILURE)

Aug 06 14:31:37 cloudron0 systemd[1]: unbound.service: Main process exited, code=exited, status=1/FAILURE
Aug 06 14:31:37 cloudron0 systemd[1]: unbound.service: Failed with result 'exit-code'.
Aug 06 14:31:38 cloudron0 systemd[1]: unbound.service: Service hold-off time over, scheduling restart.
Aug 06 14:31:38 cloudron0 systemd[1]: unbound.service: Scheduled restart job, restart counter is at 5.
Aug 06 14:31:38 cloudron0 systemd[1]: Stopped Unbound DNS Resolver.
Aug 06 14:31:38 cloudron0 systemd[1]: unbound.service: Start request repeated too quickly.
Aug 06 14:31:38 cloudron0 systemd[1]: unbound.service: Failed with result 'exit-code'.
Aug 06 14:31:38 cloudron0 systemd[1]: Failed to start Unbound DNS Resolver.
root@cloudron0:/var/lib/docker# unbound-anchor -a /var/lib/unbound/root.key
root@cloudron0:/var/lib/docker# systemctl restart unbound
root@cloudron0:/var/lib/docker# systemctl status unbound
● unbound.service - Unbound DNS Resolver
   Loaded: loaded (/etc/systemd/system/unbound.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2021-08-06 14:43:53 UTC; 2s ago
  Process: 17316 ExecStart=/usr/sbin/unbound -d (code=exited, status=1/FAILURE)
 Main PID: 17316 (code=exited, status=1/FAILURE)

Aug 06 14:43:52 cloudron0 systemd[1]: unbound.service: Main process exited, code=exited, status=1/FAILURE
Aug 06 14:43:52 cloudron0 systemd[1]: unbound.service: Failed with result 'exit-code'.
Aug 06 14:43:53 cloudron0 systemd[1]: unbound.service: Service hold-off time over, scheduling restart.
Aug 06 14:43:53 cloudron0 systemd[1]: unbound.service: Scheduled restart job, restart counter is at 5.
Aug 06 14:43:53 cloudron0 systemd[1]: Stopped Unbound DNS Resolver.
Aug 06 14:43:53 cloudron0 systemd[1]: unbound.service: Start request repeated too quickly.
Aug 06 14:43:53 cloudron0 systemd[1]: unbound.service: Failed with result 'exit-code'.
Aug 06 14:43:53 cloudron0 systemd[1]: Failed to start Unbound DNS Resolver.

MySQL and Nginx are running, and the files in /home/yellowtent/appsdata are intact

Any help would be appreciated.

[girish]

@zezaung Looks like maybe uninstalling compose also nuked docker from what I can tell. The logs say docker.service: Failed at step EXEC spawning /usr/bin/dockerd . I guess that means the docker binary disappeared. Can you check if /var/lib/docker is still there? If that too is gone, we have to re-create all the containers (I can explain how to do this later).

We install docker from the upstream repos directory, so download the 3 debs below and install them using apt install -y . docker_version below is 20.10.3 and ubuntu_codename is the output of lsb_release -cs

https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/containerd.io_1.4.3-1_amd64.deb
https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce-cli_${docker_version}~3-0~ubuntu-${ubuntu_codename}_amd64.deb
https://download.docker.com/linux/ubuntu/dists/${ubuntu_codename}/pool/stable/amd64/docker-ce_${docker_version}~3-0~ubuntu-${ubuntu_codename}_amd64.deb

[girish]

If it helps, the relevant code we use is here - https://git.cloudron.io/cloudron/box/-/blob/master/scripts/installer.sh#L77 (but you cannot run the installer script directly since it's meant to be called from a new box update).

[ZeZaung]

@girish I can't thank you enough. Had some difficulties with the -o /tmp/filename.deb (it wasn't saving the file) so I downloaded and renamed the .deb onto my computer, filezilla'd to my home directory, then used mv filename.deb /tmp

I restarted unbound and everything started running perfectly

Here is the result:
The lib files seemed untouched

root@cloudron0:/var/lib/docker# ls -al
total 200
drwx--x--x  14 root root  4096 Aug  6 14:31  .
drwxr-xr-x  47 root root  4096 Dec 10  2020  ..
drwx------   2 root root  4096 Dec 10  2020  builder
drwx--x--x   4 root root  4096 Mar 15 03:35  buildkit
drwx-----x  59 root root 12288 Aug  6 13:32  containers
drwx------   3 root root  4096 Dec 10  2020  image
drwxr-x---   3 root root  4096 Dec 10  2020  network
drwx-----x 507 root root 73728 Aug  6 13:57  overlay2
drwx------   4 root root  4096 Dec 10  2020  plugins
drwx------   2 root root  4096 Aug  6 13:57  runtimes
drwx------   2 root root  4096 Dec 10  2020  swarm
drwx------   2 root root  4096 Aug  6 13:57  tmp
drwx------   2 root root  4096 Dec 10  2020  trust
drwx-----x 458 root root 61440 Aug  6 13:57  volumes
-rw-r--r--   1 root root  1394 Aug  6 14:31 'ystemctl status unbound'

I hope this worked the way it should:

root@cloudron0:/tmp# apt install -y /tmp/containerd.deb
Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'containerd.io' instead of '/tmp/containerd.deb'
The following packages were automatically installed and are no longer required:
  amd64-microcode aufs-tools cgroupfs-mount golang-docker-credential-helpers intel-microcode iucode-tool libsecret-1-0
  libsecret-common pigz python-asn1crypto python-backports.ssl-match-hostname python-cached-property python-certifi
  python-cffi-backend python-chardet python-cryptography python-docker python-dockerpty python-dockerpycreds
  python-docopt python-enum34 python-funcsigs python-functools32 python-idna python-ipaddress python-jsonschema
  python-mock python-openssl python-pbr python-pkg-resources python-requests python-six python-texttable
  python-urllib3 python-websocket python-yaml
Use 'apt autoremove' to remove them.
The following NEW packages will be installed:
  containerd.io
0 upgraded, 1 newly installed, 0 to remove and 43 not upgraded.
After this operation, 129 MB of additional disk space will be used.
Get:1 /tmp/containerd.deb containerd.io amd64 1.4.3-1 [28.1 MB]
Selecting previously unselected package containerd.io.
(Reading database ... 151160 files and directories currently installed.)
Preparing to unpack /tmp/containerd.deb ...
Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'docker-ce-cli' instead of '/tmp/docker-ce-cli.deb'
docker-ce-cli is already the newest version (5:20.10.3~3-0~ubuntu-bionic).
The following packages were automatically installed and are no longer required:
  amd64-microcode aufs-tools cgroupfs-mount golang-docker-credential-helpers intel-microcode iucode-tool libsecret-1-0
  libsecret-common pigz python-asn1crypto python-backports.ssl-match-hostname python-cached-property python-certifi
  python-cffi-backend python-chardet python-cryptography python-docker python-dockerpty python-dockerpycreds
  python-docopt python-enum34 python-funcsigs python-functools32 python-idna python-ipaddress python-jsonschema
  python-mock python-openssl python-pbr python-pkg-resources python-requests python-six python-texttable
  python-urllib3 python-websocket python-yaml
Use 'apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 43 not upgraded.
root@cloudron0:/tmp# apt install -y /tmp/docker.deb
Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'docker-ce' instead of '/tmp/docker.deb'
The following packages were automatically installed and are no longer required:
  amd64-microcode golang-docker-credential-helpers intel-microcode iucode-tool libsecret-1-0 libsecret-common
  python-asn1crypto python-backports.ssl-match-hostname python-cached-property python-certifi python-cffi-backend
  python-chardet python-cryptography python-docker python-dockerpty python-dockerpycreds python-docopt python-enum34
  python-funcsigs python-functools32 python-idna python-ipaddress python-jsonschema python-mock python-openssl
  python-pbr python-pkg-resources python-requests python-six python-texttable python-urllib3 python-websocket
  python-yaml
Use 'apt autoremove' to remove them.
Recommended packages:
  docker-ce-rootless-extras
The following NEW packages will be installed:
  docker-ce
0 upgraded, 1 newly installed, 0 to remove and 43 not upgraded.
After this operation, 121 MB of additional disk space will be used.
Get:1 /tmp/docker.deb docker-ce amd64 5:20.10.3~3-0~ubuntu-bionic [24.8 MB]
Selecting previously unselected package docker-ce.
(Reading database ... 151174 files and directories currently installed.)
Preparing to unpack /tmp/docker.deb ...
Unpacking docker-ce (5:20.10.3~3-0~ubuntu-bionic) ...
Setting up docker-ce (5:20.10.3~3-0~ubuntu-bionic) ...
Processing triggers for systemd (237-3ubuntu10.43) ...#############################.................................]
Processing triggers for ureadahead (0.100.0-21) ...

root@cloudron0:/tmp# systemctl status box
● box.service - Cloudron Admin
   Loaded: loaded (/etc/systemd/system/box.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2021-08-06 18:49:09 UTC; 3min 8s ago
 Main PID: 21658 (node)
    Tasks: 11 (limit: 4915)
   CGroup: /system.slice/box.service
           └─21658 node /home/yellowtent/box/box.js

Aug 06 18:49:09 cloudron0 systemd[1]: Started Cloudron Admin.
Aug 06 18:49:16 cloudron0 sudo[24775]: pam_unix(sudo:session): session opened for user root by (uid=0)
Aug 06 18:49:16 cloudron0 sudo[24775]: pam_unix(sudo:session): session closed for user root
Aug 06 18:49:16 cloudron0 sudo[24838]: pam_unix(sudo:session): session opened for user root by (uid=0)
Aug 06 18:49:16 cloudron0 sudo[24838]: pam_unix(sudo:session): session closed for user root
Aug 06 18:49:47 cloudron0 sudo[29454]: pam_unix(sudo:session): session opened for user root by (uid=0)
Aug 06 18:49:47 cloudron0 sudo[29454]: pam_unix(sudo:session): session closed for user root
root@cloudron0:/tmp# systemctl status unbound
● unbound.service - Unbound DNS Resolver
   Loaded: loaded (/etc/systemd/system/unbound.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2021-08-06 14:45:47 UTC; 4h 7min ago
 Main PID: 1346 (code=exited, status=1/FAILURE)

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
root@cloudron0:/tmp# unbound-anchor -a /var/lib/unbound/root.key
boundroot@cloudron0:/tmp# systemctl restart unbound
root@cloudron0:/tmp# systemctl status unbound
● unbound.service - Unbound DNS Resolver
   Loaded: loaded (/etc/systemd/system/unbound.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2021-08-06 18:53:01 UTC; 3s ago
 Main PID: 4506 (unbound)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/unbound.service
           └─4506 /usr/sbin/unbound -d

Aug 06 18:53:01 cloudron0 systemd[1]: Started Unbound DNS Resolver.
Aug 06 18:53:01 cloudron0 unbound[4506]: [4506:0] notice: init module 0: subnet
Aug 06 18:53:01 cloudron0 unbound[4506]: [4506:0] notice: init module 1: validator
Aug 06 18:53:01 cloudron0 unbound[4506]: [4506:0] notice: init module 2: iterator
Aug 06 18:53:01 cloudron0 unbound[4506]: [4506:0] info: start of service (unbound 1.6.7).
root@cloudron0:/tmp# systemctl status nginx
● nginx.service - nginx - high performance web server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/nginx.service.d
           └─cloudron.conf
   Active: active (running) since Fri 2021-08-06 14:45:45 UTC; 4h 7min ago
     Docs: http://nginx.org/en/docs/
 Main PID: 980 (nginx)
    Tasks: 5 (limit: 4915)
   CGroup: /system.slice/nginx.service
           ├─  980 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
           ├─29499 nginx: worker process
           ├─29500 nginx: worker process
           ├─29501 nginx: worker process
           └─29502 nginx: worker process

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
root@cloudron0:/tmp# systemctl status mysql
● mysql.service - MySQL Community Server
   Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2021-08-06 14:45:44 UTC; 4h 7min ago
 Main PID: 756 (mysqld)
    Tasks: 32 (limit: 4915)
   CGroup: /system.slice/mysql.service
           └─756 /usr/sbin/mysqld --daemonize --pid-file=/run/mysqld/mysqld.pid

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

[johannesjom]

Today I also managed to completely destroy Cloudron by installing Docker-Compose (apt install docker-compose) today. I think you should prevent exactly that during the installation.

[nebulon]

@johannesjom generally it is not supported to install other things via apt manually. There are some warnings about this when you SSH into your server, but we can't quite prevent users who have root access to their own server to install things or also change essential config files. So not sure what we can do here besides warning the user.