Unable to move docker image to external storage on Hetzner storage box

[patr1ce]

Hello,

Since I miss some space on my server and 47Gb out of 80Gb of the HD are used by docker, I follow this instructions to move docker to an external storage : https://docs.cloudron.io/storage/
Everything went well since I reboot the server : the tail -f /home/yellowtent/platformdata/logs/box.log
command shows an error saying that docker was unable to start :

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

2024-01-01T19:40:23.326Z box:platform Error: Command failed: docker network create --subnet=172.18.0.0/16 --ip-range=172.18.0.0/20 --gateway 172.18.0.1 --ipv6 --subnet=fd00:c107:d509::/64 cloudron
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

    at ChildProcess.exithandler (node:child_process:419:12)
    at ChildProcess.emit (node:events:513:28)
    at maybeClose (node:internal/child_process:1091:16)
    at ChildProcess._handle.onexit (node:internal/child_process:302:5)

My /etc/systemd/system/docker.service.d/custom.conf file contains the following code

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --log-driver=journald --exec-opt native.cgroupdriver=cgroupfs --storage-driver=overlay2 --data-root=/mnt/external/docker --experimental --ip6tables

/mnt/external is mounted at startup through the following command: rclone --allow-non-empty --allow-other --config /root/.config/rclone/rclone.conf mount Hetzner: /mnt/external

Is somebody able to explain me what I am doing wrong ?

[girish]

It seems docker is not starting. Have you checked why it is not starting using journalctl -u docker -fa ?

That aside, I don't think it's a good idea to move docker images to a storage volume. This will just make things very unstable over all since docker images have to be 100% available (and hetzner storage volumes are known to go on and off randomly). You should try instead try Hetzner volume (https://docs.hetzner.com/cloud/volumes/overview/) instead of a storage box. Also, I half suspect docker is not working because the Storage Box does not support the file system semantics/APIs required by docker to store images.

[girish]

@patr1ce said in Unable to move docker image to external storage on Hetzner storage box:

I miss some space on my server and 47Gb out of 80Gb of the HD are used by docker

btw, how many apps are you hosting? That does seem like a lot. Have you already tried docker image prune -a ?

[patr1ce]

Here is the result of a journalctl -u docker -fa

Jan 01 19:40:20 ubuntu-8gb-fsn1-1 dockerd[913]: failed to start daemon: error initializing graphdriver: driver not supported
Jan 01 19:40:20 ubuntu-8gb-fsn1-1 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Jan 01 19:40:20 ubuntu-8gb-fsn1-1 systemd[1]: docker.service: Failed with result 'exit-code'.
Jan 01 19:40:20 ubuntu-8gb-fsn1-1 systemd[1]: Failed to start Docker Application Container Engine.
Jan 01 19:40:23 ubuntu-8gb-fsn1-1 systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Jan 01 19:40:23 ubuntu-8gb-fsn1-1 systemd[1]: Stopped Docker Application Container Engine.
Jan 01 19:40:23 ubuntu-8gb-fsn1-1 systemd[1]: docker.service: Start request repeated too quickly.
Jan 01 19:40:23 ubuntu-8gb-fsn1-1 systemd[1]: docker.service: Failed with result 'exit-code'.
Jan 01 19:40:23 ubuntu-8gb-fsn1-1 systemd[1]: Failed to start Docker Application Container Engine.

I have about 30 apps running. It seemed a lot to me too so I ran a docker image prune -a and docker system prune -a but it didn't free anything which looks like OK to me since cloudron is managing it by default from what I understood. I retried it just to check and this time it seems to have worked... don't know exactly why.

Hetzner volumes are 10 times more expensive than a storage box from what I see... It sounds OK since the usage are differents but little instabilities are Ok to me. It would have to work most of the times though. How can we check if Storage box is supporting the file system semantics/APIs required by docker to store images

[girish]

@patr1ce from what I understand, it's 2-3 euros a month - https://docs.hetzner.com/cloud/volumes/overview/ . You probably need only 60GB disk.

As for storage box + docker, I think docker is bailing because it's not working. To be 100% sure, maybe you can test this setup on a separate server without Cloudron and just docker. I don't have experience with mounting CIFS volume as storage for docker images unfortunately.

[patr1ce]

Yes, you're right, if only 60Gb is enough, my server should do the job. I'll try to test docker over rclone and cifs (I think rclone is using sftp with hetzner and not cifs) and let you know

[patr1ce]

Actually it is working : here is what I did on another server :

debian@vps-xxx:~$ ls /mnt/external/docker
debian@vps-xxx:~$ sudo dockerd --data-root=/mnt/external/docker &
[1] 1329
debian@vps-xxx:~$ INFO[2024-01-03T14:54:08.931197538Z] Starting up
INFO[2024-01-03T14:54:08.934354663Z] parsed scheme: "unix"                         module=grpc
INFO[2024-01-03T14:54:08.934407896Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2024-01-03T14:54:08.934447715Z] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc 
......
debian@vps-xxx:~$ docker pull stefanwalther/docker-test
Using default tag: latest
latest: Pulling from stefanwalther/docker-test
cbdbe7a5bc2a: Pull complete
4b0cddaf9d69: Pull complete
63fc8294ef6b: Pull complete
7d5b7d51bbb9: Pull complete
cd6475f4d00f: Pull complete
707d3b90a472: Pull complete
9791dd6bcc0c: Pull complete
77ee96c264c9: Pull complete
44b8f74d83e8: Pull complete
Digest: sha256:63944bf9404e3ca13a8443770e0ccf9136ce73dc985dd0d75a4d6b4ac85869d0
Status: Downloaded newer image for stefanwalther/docker-test:latest
docker.io/stefanwalther/docker-test:latest
debian@vps-xxx:~$ ls /mnt/external/docker
buildkit  containers  image  network  overlay2  plugins  runtimes  swarm  tmp  trust  volumes
debian@vps-xxx:~$ docker run -it -p 3000:3000 stefanwalther/docker-test

> docker-test@0.4.0 start /app
> node ./src/index.js

[winster] Using transports as defined in package.json

[winster] Adding 2 transport(s) to development: Console-Dev,logstash

trace: Registering route /app/src/modules/cmd-cron/cmd-cron.routes.js
trace: Registering route /app/src/modules/cmd/cmd.routes.js
trace: Registering route /app/src/modules/default/default.routes.js
trace: Registering route /app/src/modules/health-check/health-check.routes.js
(node:18) Warning: Accessing non-existent property 'padLevels' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
info: Express server listening on port 3000

The server actually responds Hello World when I browse to http://my_ip:3000

/mnt/external is still mounted with the same command than the previous server.

[girish]

@patr1ce can you post the output of docker info ?

Also, are you running docker daemon with --exec-opt native.cgroupdriver=cgroupfs --storage-driver=overlay2 ?

If that's the case, it should work on Cloudron as well.

[patr1ce]

It seems the options --storage-driver=overlay2 makes docker unable to start when the data-root is on a mounted through sftp : adding it through the following error at startup

failed to mount overlay: invalid argument     storage-driver=overlay2

if I use it with a regular data-root, no errors show up. I guess we should conclude that this makes cloudron not compatible with sftp drive for storing docker images.