Unable to move docker image to external storage on Hetzner storage box
-
Hello,
Since I miss some space on my server and 47Gb out of 80Gb of the HD are used by docker, I follow this instructions to move docker to an external storage : https://docs.cloudron.io/storage/
Everything went well since I reboot the server : thetail -f /home/yellowtent/platformdata/logs/box.log
command shows an error saying that docker was unable to start :Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? 2024-01-01T19:40:23.326Z box:platform Error: Command failed: docker network create --subnet=172.18.0.0/16 --ip-range=172.18.0.0/20 --gateway 172.18.0.1 --ipv6 --subnet=fd00:c107:d509::/64 cloudron Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? at ChildProcess.exithandler (node:child_process:419:12) at ChildProcess.emit (node:events:513:28) at maybeClose (node:internal/child_process:1091:16) at ChildProcess._handle.onexit (node:internal/child_process:302:5)
My
/etc/systemd/system/docker.service.d/custom.conf
file contains the following code[Service] ExecStart= ExecStart=/usr/bin/dockerd -H fd:// --log-driver=journald --exec-opt native.cgroupdriver=cgroupfs --storage-driver=overlay2 --data-root=/mnt/external/docker --experimental --ip6tables
/mnt/external is mounted at startup through the following command:
rclone --allow-non-empty --allow-other --config /root/.config/rclone/rclone.conf mount Hetzner: /mnt/external
Is somebody able to explain me what I am doing wrong ?
-
It seems docker is not starting. Have you checked why it is not starting using
journalctl -u docker -fa
?That aside, I don't think it's a good idea to move docker images to a storage volume. This will just make things very unstable over all since docker images have to be 100% available (and hetzner storage volumes are known to go on and off randomly). You should try instead try Hetzner volume (https://docs.hetzner.com/cloud/volumes/overview/) instead of a storage box. Also, I half suspect docker is not working because the Storage Box does not support the file system semantics/APIs required by docker to store images.
-
-
@patr1ce said in Unable to move docker image to external storage on Hetzner storage box:
I miss some space on my server and 47Gb out of 80Gb of the HD are used by docker
btw, how many apps are you hosting? That does seem like a lot. Have you already tried
docker image prune -a
? -
Here is the result of a
journalctl -u docker -fa
Jan 01 19:40:20 ubuntu-8gb-fsn1-1 dockerd[913]: failed to start daemon: error initializing graphdriver: driver not supported Jan 01 19:40:20 ubuntu-8gb-fsn1-1 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE Jan 01 19:40:20 ubuntu-8gb-fsn1-1 systemd[1]: docker.service: Failed with result 'exit-code'. Jan 01 19:40:20 ubuntu-8gb-fsn1-1 systemd[1]: Failed to start Docker Application Container Engine. Jan 01 19:40:23 ubuntu-8gb-fsn1-1 systemd[1]: docker.service: Scheduled restart job, restart counter is at 3. Jan 01 19:40:23 ubuntu-8gb-fsn1-1 systemd[1]: Stopped Docker Application Container Engine. Jan 01 19:40:23 ubuntu-8gb-fsn1-1 systemd[1]: docker.service: Start request repeated too quickly. Jan 01 19:40:23 ubuntu-8gb-fsn1-1 systemd[1]: docker.service: Failed with result 'exit-code'. Jan 01 19:40:23 ubuntu-8gb-fsn1-1 systemd[1]: Failed to start Docker Application Container Engine.
I have about 30 apps running. It seemed a lot to me too so I ran a
docker image prune -a
anddocker system prune -a
but it didn't free anything which looks like OK to me since cloudron is managing it by default from what I understood. I retried it just to check and this time it seems to have worked... don't know exactly why.Hetzner volumes are 10 times more expensive than a storage box from what I see... It sounds OK since the usage are differents but little instabilities are Ok to me. It would have to work most of the times though. How can we check if Storage box is supporting the file system semantics/APIs required by docker to store images
-
@patr1ce from what I understand, it's 2-3 euros a month - https://docs.hetzner.com/cloud/volumes/overview/ . You probably need only 60GB disk.
As for storage box + docker, I think docker is bailing because it's not working. To be 100% sure, maybe you can test this setup on a separate server without Cloudron and just docker. I don't have experience with mounting CIFS volume as storage for docker images unfortunately.
-
Actually it is working : here is what I did on another server :
debian@vps-xxx:~$ ls /mnt/external/docker debian@vps-xxx:~$ sudo dockerd --data-root=/mnt/external/docker & [1] 1329 debian@vps-xxx:~$ INFO[2024-01-03T14:54:08.931197538Z] Starting up INFO[2024-01-03T14:54:08.934354663Z] parsed scheme: "unix" module=grpc INFO[2024-01-03T14:54:08.934407896Z] scheme "unix" not registered, fallback to default scheme module=grpc INFO[2024-01-03T14:54:08.934447715Z] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>} module=grpc ...... debian@vps-xxx:~$ docker pull stefanwalther/docker-test Using default tag: latest latest: Pulling from stefanwalther/docker-test cbdbe7a5bc2a: Pull complete 4b0cddaf9d69: Pull complete 63fc8294ef6b: Pull complete 7d5b7d51bbb9: Pull complete cd6475f4d00f: Pull complete 707d3b90a472: Pull complete 9791dd6bcc0c: Pull complete 77ee96c264c9: Pull complete 44b8f74d83e8: Pull complete Digest: sha256:63944bf9404e3ca13a8443770e0ccf9136ce73dc985dd0d75a4d6b4ac85869d0 Status: Downloaded newer image for stefanwalther/docker-test:latest docker.io/stefanwalther/docker-test:latest debian@vps-xxx:~$ ls /mnt/external/docker buildkit containers image network overlay2 plugins runtimes swarm tmp trust volumes debian@vps-xxx:~$ docker run -it -p 3000:3000 stefanwalther/docker-test > docker-test@0.4.0 start /app > node ./src/index.js [winster] Using transports as defined in package.json [winster] Adding 2 transport(s) to development: Console-Dev,logstash trace: Registering route /app/src/modules/cmd-cron/cmd-cron.routes.js trace: Registering route /app/src/modules/cmd/cmd.routes.js trace: Registering route /app/src/modules/default/default.routes.js trace: Registering route /app/src/modules/health-check/health-check.routes.js (node:18) Warning: Accessing non-existent property 'padLevels' of module exports inside circular dependency (Use `node --trace-warnings ...` to show where the warning was created) info: Express server listening on port 3000
The server actually responds
Hello World
when I browse to http://my_ip:3000/mnt/external is still mounted with the same command than the previous server.
-
It seems the options --storage-driver=overlay2 makes docker unable to start when the data-root is on a mounted through sftp : adding it through the following error at startup
failed to mount overlay: invalid argument storage-driver=overlay2
if I use it with a regular data-root, no errors show up. I guess we should conclude that this makes cloudron not compatible with sftp drive for storing docker images.
-