Docker volumes are filling up disk

[archos]

Hi,
I'm running a Cloudron server, and I've run into an issue where the disk has filled up unexpectedly.

A few days ago, I had about 600 GB of free space. Now the disk is almost completely full.

I ran:

sudo docker volume prune -a
WARNING! This will remove all local volumes not used by at least one container.
Deleted Volumes:
...

Total reclaimed space: 180.5kB

Then I checked disk usage with:

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          55        55        58.44GB   2.436GB (4%)
Containers      183       142       0B        0B
Local Volumes   306       306       488.9GB   0B (0%)

sudo docker image ls
REPOSITORY                                                  TAG                  IMAGE ID       CREATED        SIZE
cloudron/com.invoiceninja.cloudronapp2                      202508020843450000   e0d6560cdcd4   21 hours ago   3.98GB
cloudron/net.freescout.cloudronapp                          202508020519350000   f86fc786326a   24 hours ago   2.5GB
cloudron/com.postiz.cloudronapp                             202508020422250000   c04fdb91ab5f   25 hours ago   7.04GB
cloudron/stirlingpdf.frooodle.cloudronapp                   202508011103200000   fa1bd83c04b4   41 hours ago   4.68GB
cloudron/com.collaboraoffice.coudronapp                     202508010420210000   9578e07a9325   2 days ago     4.25GB
cloudron/org.nodebb.cloudronapp                             202508010220250000   827b864eb257   2 days ago     3.23GB
cloudron/org.wallabag.cloudronapp2                          202507311220150000   292a70c36775   2 days ago     2.84GB
cloudron/com.bookstackapp.cloudronapp                       202507310419350000   b2cb4129b364   3 days ago     2.54GB
cloudron/im.riot.cloudronapp                                202507310023390000   b7bf0b869efe   3 days ago     2.59GB
cloudron/com.github.bitwardenrs                             202507311232500000   ed891f208de3   3 days ago     3.99GB
cloudron/com.rssbridgeapp.cloudronapp                       202507300926130000   38a6493d37e9   3 days ago     2.44GB
cloudron/sh.ntfy.cloudronapp                                202507300816310000   c16799132d8d   3 days ago     2.65GB
cloudron/org.navidrome.cloudronapp                          202507300809070000   14245290bfcc   3 days ago     2.49GB
cloudron/org.joplinapp.cloudron                             202507291200530000   234068670a9d   4 days ago     6.7GB
cloudron/lamp.cloudronapp.php74                             202507300220220000   fc222069fd95   4 days ago     2.86GB
cloudron/is.umami.cloudronapp                               202507281202140000   2c06c3e7a1d9   5 days ago     4.3GB
cloudron/info.privatebin.cloudronapp                        202507281201060000   ed3f94673baf   5 days ago     2.44GB
cloudron/io.github.ascimoo.searx                            202507272219360000   c2142ec9cfd0   6 days ago     2.69GB
cloudron/com.mirotalkp2p.cloudronapp                        202507270021030000   fc3969ebf233   7 days ago     2.64GB
cloudron/app.miniflux.cloudronapp                           202507262320310000   5874db8706d0   7 days ago     2.46GB
cloudron/org.ampache.cloudronapp                            202507240740170000   a61eb0c5f3f8   9 days ago     2.66GB
cloudron/org.joinpeertube.cloudronapp                       202507220939510000   34d2928f1ff2   11 days ago    5.91GB
cloudron/com.github.trilium.cloudronapp                     202507220522160000   3b4c9f77afbb   12 days ago    2.64GB
cloudron/io.mealie.cloudronapp                              202507212320350000   e06a4786bb4e   12 days ago    4.44GB
cloudron/app.linkwarden.cloudronapp                         202507210220020000   24b65f34c835   13 days ago    5.87GB
cloudron/org.castopod.cloudronapp                           202507181258220000   06faafbfd290   2 weeks ago    3.86GB
cloudron/calibreweb.janeczku.github                         202507181245020000   0eefd6235b3c   2 weeks ago    3.77GB
cloudron/org.wordpress.unmanaged.cloudronapp                202507160120530000   6e08ddc133c9   2 weeks ago    2.53GB
cloudron/org.wordpress.cloudronapp                          202507160120340000   a5ce17d8c897   2 weeks ago    2.6GB
cloudron/io.gitea.cloudronapp                               202507150932480000   c6c13ae03a5e   2 weeks ago    3.13GB
cloudron/io.vikunja.cloudronapp                             202507140812380000   1d564e66a8a5   2 weeks ago    2.5GB
cloudron/nu.sogo.cloudronapp2                               202507130619290000   ffa1e7af24e2   2 weeks ago    2.74GB
cloudron/com.nextcloud.cloudronapp                          202507110222030000   a0a149aa1e15   3 weeks ago    4.16GB
cloudron/co.rallly.cloudronapp                              202507091349350000   e1c0c5472c43   3 weeks ago    5.83GB
cloudron/app.listmonk.cloudronapp                           202507070220360000   71253b80857d   3 weeks ago    2.46GB
cloudron/louislam.uptimekuma.app                            202507041324150000   d94a27794d48   4 weeks ago    4.33GB
cloudron/link.linkding.cloudronapp                          202506200219300000   0f789298770d   6 weeks ago    3.7GB
cloudron/org.pixelfed.cloudronapp                           202506111207410000   dde26264a464   6 weeks ago    2.67GB
cloudron/net.roundcube.cloudronapp                          202506020019490000   b6ee6b144b65   2 months ago   2.46GB
cloudron/app.immich.cloudronapp                             202505280842390000   09f2722e8a4a   2 months ago   7.11GB
cloudron/fr.cryptpad.cloudronapp                            202505271551560000   555e16849465   2 months ago   5.79GB
cloudron/io.cloudron.surfer                                 202505161315560000   0f7872fd6357   2 months ago   2.63GB
cloudron/com.gitlab.nulide.findmydeviceserver.cloudronapp   202505141517140000   b49ee93b23e4   2 months ago   3.48GB
registry.docker.com/cloudron/postgresql                     6.0.4                98c61c4f1dcb   3 months ago   3.61GB
cloudron/org.tt_rss.cloudronapp                             202504272220020000   85d8b5936d01   3 months ago   2.49GB
cloudron/org.wordpress.unmanaged.cloudronapp                202504160121090000   ce6704524a05   3 months ago   2.53GB
cloudron/io.hackmd.cloudronapp                              202504100021510000   c190750866dd   3 months ago   3.48GB
cloudron/net.pairdrop.cloudronapp                           202503280802580000   c432a295e20b   4 months ago   2.48GB
registry.docker.com/cloudron/sftp                           3.9.1                fa80a0c95871   5 months ago   2.47GB
registry.docker.com/cloudron/graphite                       3.5.0                9a56e8d9743c   5 months ago   2.58GB
registry.docker.com/cloudron/mail                           3.15.0               2397ad1d166a   5 months ago   3.36GB
registry.docker.com/cloudron/mysql                          3.5.0                2aa20688da1e   5 months ago   2.76GB
registry.docker.com/cloudron/turn                           1.8.0                f5cf051fd9f4   5 months ago   2.44GB
registry.docker.com/cloudron/redis                          3.6.0                173b338085ce   5 months ago   2.44GB
registry.docker.com/cloudron/mongodb                        6.1.0                a42f6d9ab642   5 months ago   2.76GB

Is this normal behavior on Cloudron?
It seems like some app volumes are unnecessarily large – even for apps that don't store much data. Could this be leftover data from app updates?

Is there any safe way to clean this up?

Thanks in advance for your help!

[archos]

Largest Docker volumes on the system

437G    /var/lib/docker/volumes/2cd35318182d2ca6e0a83bfa4967707b33201c9d0389b54dc67b1da13727ece3
3.3G    /var/lib/docker/volumes/21ae2f0cdd518b6beb761b3d8f690573b88854456b1f9b8b62a59533950a36b0
2.8G    /var/lib/docker/volumes/69b8db67d33f9b732414a2a903b2c31b5af7900bed64ae1ba15a36361672c724
2.2G    /var/lib/docker/volumes/578df1d49aeae3d49ec7eb0d1b3fe247d8485e1b6bc007dc39e9bb2492ea2dbf
1.9G    /var/lib/docker/volumes/18659aaf1c1a71f000167fabeb5de42147b417840968edf857b0a28d49c059c5
1.8G    /var/lib/docker/volumes/d47e05be1ab7c9afebc44a7472b51c9123abfc0cb8dbee90efdde6f986b2b10a
1.3G    /var/lib/docker/volumes/c48e870f9326a2420d3484f9a8c9d6e101ed1b43155a072f680d4dbeb8c6d6dc
1.2G    /var/lib/docker/volumes/92ceda1eef40d4331bdc0b40dee90fa244c6322f709b17c9c057e2c23b27f7e3
879M    /var/lib/docker/volumes/4835a5f6bd07bc0ccbc48d9984188ea4cba19827c38475ff96379428fe8573e5
766M    /var/lib/docker/volumes/c52c6083cac12f83f5d1c7ac3844d29a8aa196607821d594fcea31c6efcca544
734M    /var/lib/docker/volumes/7a4d464132751933477aaaea0ac84f5b9d8c6675f14740b3bd52e68cf11a71b6
716M    /var/lib/docker/volumes/fe8398266b2ee41d4936326db1d148110c6c0246d2d55abb56ecd779982c40a7
660M    /var/lib/docker/volumes/3c73c37c138981b00cfbde782127f91876859659fe22e5795909930e44fa46ff
592M    /var/lib/docker/volumes/a7529bb1adcf3c008bb991b4b8d53071b1b978d78fc1dbf4d41400f1c2136583
573M    /var/lib/docker/volumes/c07bb67d0c6d2f7ac5f6b16379d2217fc04342bdbe50d647d8f2e348824952fe
439M    /var/lib/docker/volumes/1783e3b02509013f9e5967d24cb0308c429d316e7bf1fe91319d1d13c3d52d5f
370M    /var/lib/docker/volumes/fad8b9e0768542ebb33f3c43f37a5ab4d38bf78396f9fcbcbc26eccfaf599b2f
351M    /var/lib/docker/volumes/789e68c10bd2281d2728948d03121939e7465bca108d5fb0f4b3a4b637e57507
191M    /var/lib/docker/volumes/fdf7c2c7d257742135a5a02ff6326e12b08f990d8c68a043778a99022f45debe
177M    /var/lib/docker/volumes/c434bfac6cc7eb90390082546dfdbb95f7c9d516e69dff2f4a3f0d8d28bfadd0

The biggest volume alone is 437 GB.

I checked the contents of that volume and it’s filled with thousands of magick-* files

magick-00nzf1efCQtQKRL2bCCzds2a7Kt6cLIK

I'm not sure if these magick-* files can be safely deleted.
They look like leftover temp files from ImageMagick, but I want to make sure before removing ~400 GB of them.
Any guidance or confirmation would be appreciated.

[timconsidine]

I seem to recall there is/was a vulnerability which exposed something like this.
Dredging memory banks - maybe you can search for something like this - my searches aren’t returning anything - sorry.

[archos]

I tried docker inspect. Looks like it's a WordPress instance from a single user. Shouldn't WordPress clean this up on its own? There are thousands of magick-*.txt files taking up hundreds of GB. I think the oldest one is from around June 16th.

[archos]

@timconsidine Thanks! I’ll try to search around a bit more with that in mind. Good to know it might not be just a one-off issue.

[jdaviescoates]

Sounds to me like they are likely tmp files that can all be deleted https://stackoverflow.com/questions/35550850/lot-of-temp-magick-files-created-in-temporary-folder#39446027

[archos]

I tried deleting the files gradually starting from the oldest ones, and everything seems fine so far. Looks like they were really just leftover tmp files.

[archos]

Unfortunately, I just checked and the files are back again – only two for now. It seems something is still generating them.

[joseph]

Is it still WordPress and imagemagick temp files?

[archos]

@joseph said in Docker volumes are filling up disk:

Is it still WordPress and imagemagick temp files?

Yes, still in the same location and same naming pattern – seems to be temporary files from WordPress using ImageMagick.

[joseph]

An idea is put a cronjob into the WP app to clean up /tmp .

[p44]

@joseph I don't know if this can help, but I added to Wp cron this command:

55 1,13 * * * rm -rf /app/data/public/wp-content/cache/* > /dev/null 2>&1

You can extend this also to /tmp folders.

I added this line to save space before backup. You can edit this according to backup scheduled times

[SansGuidon]

Is it related to a plugin? I know that by default wp-cron jobs were not always running in WP so I had to force create a cron job to run wp-cron on a schedule
It is supposed to be the reponsibility of WP Plugins to cleanup after them, by using the wp-cron, but it is by default disabled afaik. So what I have scheduled for a long time in Cloudron is */30 * * * * wget -q -O - https://wp.xxx.xxx/wp-cron.php?doing_wp_cron >/dev/null 2>&1

[archos]

Thanks a lot everyone for the advice
I’ve added some cleanup jobs to the WP cron for now.
I’ll let you know if it helps.

[p44]

@SansGuidon Cloudron cron jobs works perfectly with Wordpress...

@archos Keep us posted

[SansGuidon]

@p44 said in Docker volumes are filling up disk:

@SansGuidon Cloudron cron jobs works perfectly with Wordpress...

@archos Keep us posted

Glad it works for you. WP would constantly complaining those WP-cron jobs were disabled (see Cloudron docs, they mention wp-cron is disabled https://docs.cloudron.io/packages/wordpress-managed/ ), so I came with the fix I mention to get rid of the warnings/errors shown in my WP instance maybe my solution is outdated yet it runs for so long now.

[p44]

@SansGuidon Mentioned guide also say "To add custom cron events, use Cloudron's built-in cron or use a plugin like WP Crontrol." And it works like a sharm...

Before Cloudron I had a lot of problems publishing scheduled posts because WP Cron is activated only when user visits the website. Cloudron guide say "WordPress' built-in cron task schedule wp-cron is disabled since it is not effective for low traffic websites."

[SansGuidon]

@p44 said in Docker volumes are filling up disk:

@SansGuidon Mentioned guide also say "To add custom cron events, use Cloudron's built-in cron or use a plugin like WP Crontrol." And it works like a sharm...

Before Cloudron I had a lot of problems publishing scheduled posts because WP Cron is activated only when user visits the website. Cloudron guide say "WordPress' built-in cron task schedule wp-cron is disabled since it is not effective for low traffic websites."

Interesting to know, however what I remember is that my site would appear non healthy without forcing some cron jobs to trigger, so I guess that's why I had to add this line in my cron tab, despite Cloudron is supposed to make that not necessary. Weird. Anyway if that works like a charm for everyone but me, I guess I might have different needs

[p44]

@SansGuidon It still does not work for you?

[archos]

The cron job is working perfectly, temporary files are no longer accumulating. Many thanks to everyone for the advice.