Moving docker containers crashes apps

[enesaydogan]

I used this instructions on cloudron docs. Apps not working after starting docker or rebooting machine. docker ps has no container running.

Moving docker images to another location
Cloudron uses Docker for containerizing applications and docker images tend to consume a lot >of space. The docker images are stored by default at /var/lib/docker. They can be moved to an >external storage as follows:

Stop docker
systemctl stop docker

Move existing contents to external storage. Any external storage must be formatted as ext4. >Assuming, /mnt/docker is the new location:
rsync -aHSX /var/lib/docker /mnt/docker

Link /var/lib/docker to the new location
rm -rf /var/lib/docker
ln -s /mnt/docker /var/lib/docker

Alternately, you can bind mount mount --bind /mnt/docker /var/lib/docker. This, however, >requires adding an entry in /etc/fstab to persist across reboots.

Start docker
systemctl start docker

I get this error on docker log :

rd process" pid=1564
489c9c1b21a6d78b5c538cd395416ec50f88 version=v1.0.3
" module=containerd type=io.containerd.content.v1
.." module=containerd type=io.containerd.snapshotter.v1
1.btrfs" error="path /mnt/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module=containerd
fs"..." module=containerd type=io.containerd.snapshotter.v1
module=containerd type=io.containerd.metadata.v1
n" error="path /mnt/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module="containerd/io.containerd.metadata.v1.bolt"
 module=containerd type=io.containerd.differ.v1
odule=containerd type=io.containerd.gc.v1
" module=containerd type=io.containerd.grpc.v1
odule=containerd type=io.containerd.grpc.v1
le=containerd type=io.containerd.grpc.v1
dule=containerd type=io.containerd.grpc.v1
." module=containerd type=io.containerd.grpc.v1
dule=containerd type=io.containerd.grpc.v1
dule=containerd type=io.containerd.grpc.v1
" module=containerd type=io.containerd.grpc.v1
 module=containerd type=io.containerd.grpc.v1
" module=containerd type=io.containerd.monitor.v1
module=containerd type=io.containerd.runtime.v1
ule=containerd type=io.containerd.grpc.v1
odule=containerd type=io.containerd.grpc.v1
..." module=containerd type=io.containerd.grpc.v1
-containerd-debug.sock" module="containerd/debug"
-containerd.sock" module="containerd/grpc"
containerd
took 0.00 seconds"
period"
runtime"

[nebulon]

I haven't seen this error before. If you have backups of all the apps, you could try to start with an empty /var/lib/docker folder and restore all apps from the Cloudron dashboard, this would hopefully refetch all images and rebuild the app containers as well.

Also which Cloudron version are you using and just to be sure that docker was not somehow updated by the system automatically, which docker version is this?

[enesaydogan]

I dont update manualy as instructed by cloudron. Luckily i do use digitalocean snapshots before doing this kinda of stuff. I am not good with this stuff now i am scared to try again. My intention was the inform you guys about the problem maybe docs out of date or may be i am missing something. Increased disk size on vm so i dont have do this for now. Thanks for answer.

Cloudron version : v2.4.1

root@my:~# docker info
Containers: 18
 Running: 15
 Paused: 0
 Stopped: 3
Images: 12
Server Version: 18.03.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-130-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.953GiB
Name: my.sinanerholding.com.tr
ID: DTCD:EOBU:MMA7:YA2U:TGRX:IPPE:YG5X:ZVE5:26TX:UZQD:AFKZ:74KS
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

[girish]

@enesaydogan Indeed something seems off. The error report you pasted seems to complain about btrfs but Cloudron itself does not use btrfs. I am trying this on a DO instance myself now.

[girish]

Just tried this and it works for me.

Note that I too see the following message in my logs:

Aug 02 17:29:34 my.smartserver.space dockerd[20759]: time="2018-08-02T17:29:34Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." module=containerd type=io.containerd.snapshotter.v1
Aug 02 17:29:34 my.smartserver.space dockerd[20759]: time="2018-08-02T17:29:34Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /mnt/volume_sfo2_01/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module=containerd
Aug 02 17:29:34 my.smartserver.space dockerd[20759]: time="2018-08-02T17:29:34Z" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /mnt/volume_sfo2_01/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module="containerd/io.containerd.metadata.v1.bolt"

But the above error is harmless. It is just saying that for btrfs plugin we need btrfs volume. This can be ignored since Cloudron does not use btrfs plugin.