Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    Solved /var/lib/docker/ is filling up /dev/sda2 primary partition?

    Cubby
    2
    18
    150
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • N
      natzilla last edited by natzilla

      23d973b4-16a9-425a-ac04-f41fa2470349-image.png

      This seems rather large for 8 apps installed with anything using large files moved onto a different mount volume.

      It does not let me into this directory to see what specifically is taking up all this space
      7d588f8c-090f-4500-bf5b-2fc802973164-image.png

      I am a little concerned that if this continues my primary partition will lock up and everything will stop working soon.
      073fa44b-7116-43f5-b0c1-548bef63e6b4-image.png

      1 Reply Last reply Reply Quote 0
      • nebulon
        nebulon Staff last edited by

        We have some information on the disk layout for Cloudron at https://docs.cloudron.io/storage/ there you can also find information on how to relocate the docker images to another disk.

        Looking at the huge "Everything else" is it maybe possible that you were previously using backups on the local disk? Can you check the size of /var/backups ?

        N 1 Reply Last reply Reply Quote 0
        • N
          natzilla @nebulon last edited by

          @nebulon

          It was never using local backups to begin with.
          33b93acc-5caf-4487-85ed-d019fa248ef7-image.png
          09352376-376b-427b-baea-e1422609f9f7-image.png
          it does still have some data, but 200 gigs, it does not.

          I was able to get into the docker folder and check to see what was taking up the space. It was all focused on one folder volumes/d2e397a84d2e73d5a8ce97860a01e4d6c4929e6a25fcff0af02c316b0a82d981/_data/

          It seemed to contained large files that I transferred onto the server to then move somewhere else. I used cubby for this, so that makes me wonder if cubby retains what is uploaded to it in the docker folder for some reason?

          For my uploads I would upload to cubby, then go into the file manager of cloudron to cut paste where I wanted the files.

          I don't know if that is a bug or not but I at least found the cause of it by removing everything in there.
          89f006ea-b6b4-4454-a350-55a77d20bc75-image.png

          1 Reply Last reply Reply Quote 1
          • nebulon
            nebulon Staff last edited by

            Have you used the platform feature to move the cubby application data or were you moving things via SSH?

            cubby as such should be fine when the files are moved or changed behind its back (like in your case with the cloudron filemanager) so I don't think this would interfere here.

            Further, the unix tools like du are not working properly on those virtual filesystems which docker creates under /var/lib/docker for the mounts and volumes, so I am not sure what really happens if you remove files there via SSH. To be honest that sounds a bit risky.

            N 2 Replies Last reply Reply Quote 0
            • N
              natzilla @nebulon last edited by natzilla

              @nebulon
              I was not moving anything with ssh. At first cubby was default installed (not moved over to my second partition) I would upload there then use this page to move it.
              0f0afafa-bd8f-49eb-9889-d095d15c81b7-image.png

              Later yesterday is when I noticed it was still on that primary partition and it was getting full. so I started a data directory change.

              464a963f-482e-4312-9be8-ed6994904563-image.png

              The partition still stayed full after the directory change.

              1 Reply Last reply Reply Quote 0
              • N
                natzilla @nebulon last edited by

                @nebulon
                I've done another test now

                I had to reinstall cubby as the file deletion I did broke the upload function on cubby.
                First thing I did was move cubby to the second partition before any upload.

                e02869b4-7602-4b8e-a412-41ca48782fa4-image.png
                Before

                After
                859efa64-18c2-47d6-ae92-7340d8798149-image.png
                We can see it increased on the primary partition

                Now I moved them over to another location off app, and the space still shows being used by those uploads.

                checking the docker folder again I can see it's retaining the files I moved off app

                4a81ecc6-27fe-4fa7-9647-26aa70859eb3-image.png

                So it seems as if this might be a bug?

                nebulon 1 Reply Last reply Reply Quote 0
                • nebulon
                  nebulon Staff @natzilla last edited by

                  @natzilla I wonder what that volume is, maybe this is the /tmp volume from the app? I don't think I have implemented the cleanup of files after upload as eventually they will get removed from /tmp automatically, I have to check though what the exact behavior there is.

                  Can you maybe run docker ps, then find the container ID of that cubby instance and then run docker inspect <containerid> and check what that volume is used for there?

                  N 1 Reply Last reply Reply Quote 0
                  • N
                    natzilla @nebulon last edited by

                    @nebulon

                    There are two containers

                    f391e386-6e82-4e70-97da-1f58a4d6d556-image.png

                    and

                    3caf49e8-c2f9-4ddf-a35e-e390cd6cfe7c-image.png

                    containers

                    2a5715d3-856a-4a79-827b-c6069811aa66-image.png

                    nebulon 1 Reply Last reply Reply Quote 0
                    • nebulon
                      nebulon Staff @natzilla last edited by

                      @natzilla the second one is the sftp service container from the platform, that is fine, but it also does not reference that volume in question (the volume with ID starting with "8aa2b....") for that please run docker inspect <appcontainerid> and check what that volume is mounted into in the container.

                      N 1 Reply Last reply Reply Quote 0
                      • N
                        natzilla @nebulon last edited by

                        @nebulon

                        I did run that one. It's the first image in my last reply. We can see 8aa2 is listed there.

                        nebulon 1 Reply Last reply Reply Quote 0
                        • nebulon
                          nebulon Staff @natzilla last edited by

                          @natzilla can you run this without the | grep volume as it is a json file and the lines around that line with the volume id will indicate for what it is used.

                          N 1 Reply Last reply Reply Quote 0
                          • N
                            natzilla @nebulon last edited by natzilla

                            @nebulon

                            As requested

                            root@pon3-server:/var/lib/docker/volumes/8aa2bbafb0b8d278a3af3d83c71c68e8643cbb22ce849050c6cd2fb3a45bf30c/_data# docker inspect bb17d9e09952
                            [
                            {
                            "Id": "bb17d9e0995289960d36072be40786b7efda022c2ee8af80d382d78586a38d75",
                            "Created": "2022-12-14T15:29:16.192977943Z",
                            "Path": "/app/code/start.sh",
                            "Args": [],
                            "State": {
                            "Status": "running",
                            "Running": true,
                            "Paused": false,
                            "Restarting": false,
                            "OOMKilled": false,
                            "Dead": false,
                            "Pid": 107746,
                            "ExitCode": 0,
                            "Error": "",
                            "StartedAt": "2022-12-14T15:29:16.794542023Z",
                            "FinishedAt": "0001-01-01T00:00:00Z"
                            },
                            "Image": "sha256:61e1032f9a2bbb21c35ce19efbc2b232a774c49c66216c4064a1db2b58ee1962",
                            "ResolvConfPath": "/var/lib/docker/containers/bb17d9e0995289960d36072be40786b7efda022c2ee8af80d382d78586a38d75/resolv.conf",
                            "HostnamePath": "/var/lib/docker/containers/bb17d9e0995289960d36072be40786b7efda022c2ee8af80d382d78586a38d75/hostname",
                            "HostsPath": "/var/lib/docker/containers/bb17d9e0995289960d36072be40786b7efda022c2ee8af80d382d78586a38d75/hosts",
                            "LogPath": "",
                            "Name": "/aa3aad90-42e4-4086-bfc4-ff806b12e78a",
                            "RestartCount": 0,
                            "Driver": "overlay2",
                            "Platform": "linux",
                            "MountLabel": "",
                            "ProcessLabel": "",
                            "AppArmorProfile": "docker-cloudron-app",
                            "ExecIDs": null,
                            "HostConfig": {
                            "Binds": null,
                            "ContainerIDFile": "",
                            "LogConfig": {
                            "Type": "syslog",
                            "Config": {
                            "syslog-address": "udp://127.0.0.1:2514",
                            "syslog-format": "rfc5424",
                            "tag": "aa3aad90-42e4-4086-bfc4-ff806b12e78a"
                            }
                            },
                            "NetworkMode": "cloudron",
                            "PortBindings": {},
                            "RestartPolicy": {
                            "Name": "unless-stopped",
                            "MaximumRetryCount": 0
                            },
                            "AutoRemove": false,
                            "VolumeDriver": "",
                            "VolumesFrom": null,
                            "CapAdd": [],
                            "CapDrop": [
                            "NET_RAW"
                            ],
                            "CgroupnsMode": "host",
                            "Dns": [
                            "172.18.0.1"
                            ],
                            "DnsOptions": null,
                            "DnsSearch": [
                            "."
                            ],
                            "ExtraHosts": null,
                            "GroupAdd": null,
                            "IpcMode": "private",
                            "Cgroup": "",
                            "Links": null,
                            "OomScoreAdj": 0,
                            "PidMode": "",
                            "Privileged": false,
                            "PublishAllPorts": false,
                            "ReadonlyRootfs": true,
                            "SecurityOpt": [
                            "apparmor=docker-cloudron-app"
                            ],
                            "UTSMode": "",
                            "UsernsMode": "",
                            "ShmSize": 67108864,
                            "Runtime": "runc",
                            "ConsoleSize": [
                            0,
                            0
                            ],
                            "Isolation": "",
                            "CpuShares": 512,
                            "Memory": 483393536,
                            "NanoCpus": 0,
                            "CgroupParent": "",
                            "BlkioWeight": 0,
                            "BlkioWeightDevice": null,
                            "BlkioDeviceReadBps": null,
                            "BlkioDeviceWriteBps": null,
                            "BlkioDeviceReadIOps": null,
                            "BlkioDeviceWriteIOps": null,
                            "CpuPeriod": 0,
                            "CpuQuota": 0,
                            "CpuRealtimePeriod": 0,
                            "CpuRealtimeRuntime": 0,
                            "CpusetCpus": "",
                            "CpusetMems": "",
                            "Devices": null,
                            "DeviceCgroupRules": null,
                            "DeviceRequests": null,
                            "KernelMemory": 0,
                            "KernelMemoryTCP": 0,
                            "MemoryReservation": 0,
                            "MemorySwap": 536870912,
                            "MemorySwappiness": null,
                            "OomKillDisable": false,
                            "PidsLimit": null,
                            "Ulimits": null,
                            "CpuCount": 0,
                            "CpuPercent": 0,
                            "IOMaximumIOps": 0,
                            "IOMaximumBandwidth": 0,
                            "Mounts": [
                            {
                            "Type": "bind",
                            "Source": "/mnt/volumes/d828f00dc01d4eb5be43ca61defeb4b5/dropbox",
                            "Target": "/app/data"
                            }
                            ],
                            "MaskedPaths": [
                            "/proc/asound",
                            "/proc/acpi",
                            "/proc/kcore",
                            "/proc/keys",
                            "/proc/latency_stats",
                            "/proc/timer_list",
                            "/proc/timer_stats",
                            "/proc/sched_debug",
                            "/proc/scsi",
                            "/sys/firmware"
                            ],
                            "ReadonlyPaths": [
                            "/proc/bus",
                            "/proc/fs",
                            "/proc/irq",
                            "/proc/sys",
                            "/proc/sysrq-trigger"
                            ]
                            },
                            "GraphDriver": {
                            "Data": {
                            "LowerDir": "/var/lib/docker/overlay2/9c64364d1e56b61ec0b8054ae1264890152caed47165466e9af88b84f392d2d8-init/diff:/var/lib/docker/overlay2/2239409d9d6dd34f365e40a3f8cee52897db3b0521d0b6a6a16b3f2726dca4c3/diff:/var/lib/docker/overlay2/9d41c30e6e6bc5e47c3c02c8b49fe9b31d95fe928af3c735235caca8f85a28e6/diff:/var/lib/docker/overlay2/c4853771022610997fe96328d237d3d7fd8e05e958153b3415abe5df7de3f076/diff:/var/lib/docker/overlay2/240c4bcf910cbbc49698a04b95da6a17257807cb252f36f618358b1ae042c0c4/diff:/var/lib/docker/overlay2/ae2ec42c74ca7ff1c2121271c438a06d5f43517a021fa27fedb3de5adfd4126e/diff:/var/lib/docker/overlay2/74f204a2f887a42c80528e7dd486b39a9e7e7f18c65dbfd45f6b0528d4209434/diff:/var/lib/docker/overlay2/40eb436639d648361086d6b347f29c52513893720131ea3c357ba186aab10c53/diff:/var/lib/docker/overlay2/419d743c3a5cd0ab05182f78265564c4d04eb1fec8339ac710d53e3769794d83/diff:/var/lib/docker/overlay2/ff1cfbb2ae944f740122a19ef2dab2d43373c440eb3c7169d2943e1851b1f2b2/diff:/var/lib/docker/overlay2/8dc58f88f40e247dd3d93507fb311e96012157cb01b8e1fe9750d7708dbfc2e6/diff:/var/lib/docker/overlay2/b122bc434a62507a71aebae852e442132b8670ed9671f4a05f45ab9d07b13278/diff:/var/lib/docker/overlay2/c73304048cfc522dc8b1d46cda84e34401cb583e1f808227754bd6c5a557e6bf/diff:/var/lib/docker/overlay2/154763b48008e44bd3de952b70756cc9e2ee9c28bfad1e6b6680ff7b11f170af/diff:/var/lib/docker/overlay2/ab54942b6e3a8214edb7740c2e40ad9a7108df32900d610272901c2c51fc72b0/diff:/var/lib/docker/overlay2/cc5d3a9f9e407cfa105df870d36b4b46a07d45223f53b5cc7a4e3f7cb8348307/diff:/var/lib/docker/overlay2/c426c24563d905ce79e17964881fae9aec29e3bd60db8e1d0c4f915c916de735/diff:/var/lib/docker/overlay2/80fb267c7479664dac1d9f28c77050536144f50357fc1996dbe7cd35d438ff53/diff:/var/lib/docker/overlay2/5a25242ee7066afe2461f8be8f607d2c7d78d0ec14ee4d84655a805987c256b4/diff:/var/lib/docker/overlay2/f557a9ff958a680b079ec20ad3adcdd1e5760d20b6570474ca7556239b2d590e/diff:/var/lib/docker/overlay2/a49d9997d75315f730f1aa7d35b0bd550fd85cafd704ddb8a24f7d4deb0f7b9e/diff",
                            "MergedDir": "/var/lib/docker/overlay2/9c64364d1e56b61ec0b8054ae1264890152caed47165466e9af88b84f392d2d8/merged",
                            "UpperDir": "/var/lib/docker/overlay2/9c64364d1e56b61ec0b8054ae1264890152caed47165466e9af88b84f392d2d8/diff",
                            "WorkDir": "/var/lib/docker/overlay2/9c64364d1e56b61ec0b8054ae1264890152caed47165466e9af88b84f392d2d8/work"
                            },
                            "Name": "overlay2"
                            },
                            "Mounts": [
                            {
                            "Type": "volume",
                            "Name": "a9d5b4ebb3669ff2adc9048e3317657d9db8da14a1e86bab850a1c40464a0b57",
                            "Source": "/var/lib/docker/volumes/a9d5b4ebb3669ff2adc9048e3317657d9db8da14a1e86bab850a1c40464a0b57/_data",
                            "Destination": "/run",
                            "Driver": "local",
                            "Mode": "",
                            "RW": true,
                            "Propagation": ""
                            },
                            {
                            "Type": "bind",
                            "Source": "/mnt/volumes/d828f00dc01d4eb5be43ca61defeb4b5/dropbox",
                            "Destination": "/app/data",
                            "Mode": "",
                            "RW": true,
                            "Propagation": "rprivate"
                            },
                            {
                            "Type": "volume",
                            "Name": "9120108234b71ccd3f3a7eed56ab718e0145018a80a7a6982fc9904cf25c2792",
                            "Source": "/var/lib/docker/volumes/9120108234b71ccd3f3a7eed56ab718e0145018a80a7a6982fc9904cf25c2792/_data",
                            "Destination": "/home/cloudron/.cache",
                            "Driver": "local",
                            "Mode": "",
                            "RW": true,
                            "Propagation": ""
                            },
                            {
                            "Type": "volume",
                            "Name": "db9dcbde32d2b270f24b03c7e6820fe86ba4b6440275f2ccf623931faa992ccd",
                            "Source": "/var/lib/docker/volumes/db9dcbde32d2b270f24b03c7e6820fe86ba4b6440275f2ccf623931faa992ccd/_data",
                            "Destination": "/root/.cache",
                            "Driver": "local",
                            "Mode": "",
                            "RW": true,
                            "Propagation": ""
                            },
                            {
                            "Type": "volume",
                            "Name": "8aa2bbafb0b8d278a3af3d83c71c68e8643cbb22ce849050c6cd2fb3a45bf30c",
                            "Source": "/var/lib/docker/volumes/8aa2bbafb0b8d278a3af3d83c71c68e8643cbb22ce849050c6cd2fb3a45bf30c/_data",
                            "Destination": "/tmp",
                            "Driver": "local",
                            "Mode": "",
                            "RW": true,
                            "Propagation": ""
                            }
                            ],
                            "Config": {
                            "Hostname": "aa3aad90-42e4-4086-bfc4-ff806b12e78a",
                            "Domainname": "",
                            "User": "",
                            "AttachStdin": false,
                            "AttachStdout": false,
                            "AttachStderr": false,
                            "Tty": true,
                            "OpenStdin": false,
                            "StdinOnce": false,
                            "Env": [
                            "CLOUDRON=1",
                            "CLOUDRON_PROXY_IP=172.18.0.1",
                            "CLOUDRON_APP_HOSTNAME=aa3aad90-42e4-4086-bfc4-ff806b12e78a",
                            "CLOUDRON_WEBADMIN_ORIGIN=----------------------.net",
                            "CLOUDRON_API_ORIGIN=--------------------------.net",
                            "CLOUDRON_APP_ORIGIN=-------------------.net",
                            "CLOUDRON_APP_DOMAIN=---------------------.net",
                            "CLOUDRON_POSTGRESQL_URL=postgres://useraa3aad9042e44086bfc4ff806b12e78a:cda2df1e66dbfeb5b419e08176cb94d5927f7cceda16dae0b5d1e140be9690e6f5bc9617c5613487c0990b2f9cefadf60bd160131454a3c2336251c1c32b109c@postgresql/dbaa3aad9042e44086bfc4ff806b12e78a",
                            "CLOUDRON_POSTGRESQL_USERNAME=useraa3aad9042e44086bfc4ff806b12e78a",
                            "CLOUDRON_POSTGRESQL_PASSWORD=cda2df1e66dbfeb5b419e08176cb94d5927f7cceda16dae0b5d1e140be9690e6f5bc9617c5613487c0990b2f9cefadf60bd160131454a3c2336251c1c32b109c",
                            "CLOUDRON_POSTGRESQL_HOST=postgresql",
                            "CLOUDRON_POSTGRESQL_PORT=5432",
                            "CLOUDRON_POSTGRESQL_DATABASE=dbaa3aad9042e44086bfc4ff806b12e78a",
                            "CLOUDRON_LDAP_SERVER=172.18.0.1",
                            "CLOUDRON_LDAP_HOST=172.18.0.1",
                            "CLOUDRON_LDAP_PORT=3002",
                            "CLOUDRON_LDAP_URL=ldap://172.18.0.1:3002",
                            "CLOUDRON_LDAP_USERS_BASE_DN=ou=users,dc=cloudron",
                            "CLOUDRON_LDAP_GROUPS_BASE_DN=ou=groups,dc=cloudron",
                            "CLOUDRON_LDAP_BIND_DN=cn=aa3aad90-42e4-4086-bfc4-ff806b12e78a,ou=apps,dc=cloudron",
                            "CLOUDRON_LDAP_BIND_PASSWORD=aff4d7cf97262f6b6302803b873295880a45c3aa8f33cc8bde3efff901578391273ffb4f11168342659a0ae0dfac4ed9e35e1ef70f64f23a53a3d5d030481c96",
                            "PATH=/usr/local/node-16.13.1/bin:/usr/local/go-1.17.5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                            "DEBIAN_FRONTEND=noninteractive",
                            "GOROOT=/usr/local/go-1.17.5",
                            "CONTAINER_NAME=postgres-server-cubby",
                            "PGPASSWORD=",
                            "DEBUG=cubby*"
                            ],
                            "Cmd": [
                            "/app/code/start.sh"
                            ],
                            "Image": "cloudron/io.cloudron.cubby:20221023-213915-3426f3b61",
                            "Volumes": {
                            "/home/cloudron/.cache": {},
                            "/root/.cache": {},
                            "/run": {},
                            "/tmp": {}
                            },
                            "WorkingDir": "/app/code",
                            "Entrypoint": null,
                            "OnBuild": null,
                            "Labels": {
                            "appId": "aa3aad90-42e4-4086-bfc4-ff806b12e78a",
                            "fqdn": "----------------------.net",
                            "isCloudronManaged": "true",
                            "isSubcontainer": "false"
                            }
                            },
                            "NetworkSettings": {
                            "Bridge": "",
                            "SandboxID": "f776f5c7246279bd183a4721e1d7db762fa76ac3a9bf7fd21d255c9c9a56c659",
                            "HairpinMode": false,
                            "LinkLocalIPv6Address": "",
                            "LinkLocalIPv6PrefixLen": 0,
                            "Ports": {},
                            "SandboxKey": "/var/run/docker/netns/f776f5c72462",
                            "SecondaryIPAddresses": null,
                            "SecondaryIPv6Addresses": null,
                            "EndpointID": "",
                            "Gateway": "",
                            "GlobalIPv6Address": "",
                            "GlobalIPv6PrefixLen": 0,
                            "IPAddress": "",
                            "IPPrefixLen": 0,
                            "IPv6Gateway": "",
                            "MacAddress": "",
                            "Networks": {
                            "cloudron": {
                            "IPAMConfig": {
                            "IPv4Address": "172.18.16.166"
                            },
                            "Links": null,
                            "Aliases": [
                            "aa3aad90-42e4-4086-bfc4-ff806b12e78a",
                            "bb17d9e09952"
                            ],
                            "NetworkID": "805d05f75bbd29d71f2e6dbf9b34dd5d4fdfe81f2762742b7b576b2ac80b9cc8",
                            "EndpointID": "5ea9bcd5c560cc945f4c4f21067a21d8ee3168c7c54d93daf94322e71eae75c7",
                            "Gateway": "172.18.0.1",
                            "IPAddress": "172.18.16.166",
                            "IPPrefixLen": 16,
                            "IPv6Gateway": "fd00:c107:d509::1",
                            "GlobalIPv6Address": "fd00:c107:d509::9",
                            "GlobalIPv6PrefixLen": 64,
                            "MacAddress": "02:42:ac:12:10:a6",
                            "DriverOpts": null
                            }
                            }
                            }
                            }
                            ]
                            root@pon3-server:/var/lib/docker/volumes/8aa2bbafb0b8d278a3af3d83c71c68e8643cbb22ce849050c6cd2fb3a45bf30c/_data#

                            1 Reply Last reply Reply Quote 0
                            • nebulon
                              nebulon Staff last edited by

                              ah thanks, so it is the /tmp folder. I think it is related to the upload cache in cubby. Will try to reproduce this tomorrow and also hopefully fix it in cubby then.

                              N 1 Reply Last reply Reply Quote 0
                              • N
                                natzilla @nebulon last edited by

                                @nebulon

                                Awesome, I appreciate it.

                                1 Reply Last reply Reply Quote 0
                                • Topic has been marked as a question  nebulon nebulon 
                                • nebulon
                                  nebulon Staff last edited by

                                  Its quite busy right now here, so until the app package is fixed, it should be safe to use the webterminal and remove those files from the /tmp folder within the app to clear out some space in the meantime.

                                  N 1 Reply Last reply Reply Quote 0
                                  • N
                                    natzilla @nebulon last edited by

                                    @nebulon

                                    Acknowledged, I will do so until it's patched.

                                    1 Reply Last reply Reply Quote 0
                                    • Moved from Support by  nebulon nebulon 
                                    • nebulon
                                      nebulon Staff last edited by

                                      Fixed with cubby 1.1.0

                                      N 1 Reply Last reply Reply Quote 1
                                      • Topic has been marked as solved  nebulon nebulon 
                                      • N
                                        natzilla @nebulon last edited by

                                        @nebulon

                                        Awesome, thanks again.

                                        1 Reply Last reply Reply Quote 0
                                        • First post
                                          Last post
                                        Powered by NodeBB