docker overlay2 directory running wild: over 75GB of space used

[foliovision]

Hi Nebulon and Girish,

I hope you gentlemen are doing well. How time flies when you're having fun. It's been more than five years of Cloudron for us.

We keep running into disk space issues with our Cloudron which is running on an 80GB DigitalOcean droplet. We've cleaned up NextCloud (one can't conveniently use the file sharing because of the backups which are difficult to fully delete) and Grav (which for some reason creates dozens and dozens of backups adding up to GB.

In total we have about 14GB of data in /home/yellowtent/appsdata, with another 8GB of mongodb in /home/yellowtent/platformdata. There's some other bits and pieces like logs but these are the big ones. Yet we have less than 15GB of space on our 80GB VPS.

What's left is the overlay2 directory (/var/lib/docker/overlay2). There's 75GB! of data stored there. I've seen that you've written that some of this storage is virtual (it must be as there wouldn't be room for the OS on our VPS otherwise, let alone any other data).

Still 75GB seems excessive and there's so little usable space.

I've run commands like docker ps -a, docker volume prune, docker system prune -a, docker image prune --all. This hasn't got me far. Here's all that was deleted:

root@my:/var/lib/docker/overlay2# docker system prune -a
WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all images without at least one container associated to them
  - all build cache

Are you sure you want to continue? [y/N] y
Deleted Containers:
26796e79b1f631b60db7b03bad4f07b36d69769f73da559f080f1f25c4ef42e5
a5c57f2c3edffe486266949c6d82309aed7cc128895db305a972671c8d33ef6c
9f0dc548c88e9722014e94bb0137893fb16a690a76fc0a0f2df4c26f6e8c1055
b6b3ff484ec1306dc728a840375153b3dc2bad7fe0f2dc8e81136ce9339ae538

Total reclaimed space: 0B

1. What do we have to do to get more usable user space on our 80GB VPS?
2. How can we reduce the amount of storage used in /var/lib/docker/overlay2?
3. Data stored in our MongoDB seems to be very large. I suspect it's UptimeKuma history. Do you have any good tips on how to trim MongoDB in Cloudron down? A workflow, caveats/gotchas, safe way to reduce the MongoDB size?

Thanks in advance for your help. Please answer to the level in which I wrote. I.e. I'm an advanced user and not a programmer/full-time server admin. Please be explicit and not implicit with your instruction.

Making the web work for you, Alec

[joseph]

If you ran du on the overlay2 , then the returned size is not correct because du is not aware of overlay filesystems. You have to docker system df instead.

I thought uptimekuma uses sqlite and not mongodb ... Do you have any other app using mongodb?

[foliovision]

Hi Joseph,

Thanks for the tip about docker system df. Here's the results, which look more reasonable.

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          17        17        21.5GB    2.436GB (11%)
Containers      22        18        0B        0B
Local Volumes   36        36        1.113GB   0B (0%)
Build Cache     0         0         0B        0B

1. How do we reclaim the 2.46GB of reclaimable space?
2. Why on our 80GB Cloudron VPS do we only have about 30GB available for data? That's a lot of overhead.
3. Is there some way to organise this Cloudron VPS more efficiently? Is Cloudron stashing a day or week of backups somewhere on this VPS?

Please answer all three questions if you could.

I'll get back to you about the MongoDB issue when I have more information.

Making the web work for you, Alec

[foliovision]

Hi Joseph,

You're right, it's not UptimeKuma. RocketChat is the web app which we run which uses MongoDB. We'll work on trimming it down.

Thanks, Alec

[foliovision]

Hi Joseph,

My post above about MongoDB is not meant to close the thread. These are my three follow-up questions.

1. How do we reclaim the 2.46GB of reclaimable space?
2. Why on our 80GB Cloudron VPS do we only have about 30GB available for data? That's a lot of overhead.
3. Is there some way to organise this Cloudron VPS more efficiently? Is Cloudron stashing a day or week of backups somewhere on this VPS?

I look forward to your answers.

Cordial regards, Alec Kinnear

[necrevistonnezr]

Maybe https://forum.cloudron.io/topic/1579/var-lib-docker-overlay2-getting-full?_=1760112116784 is helpful:

@nebulon said in /var/lib/docker/overlay2 getting full:

For a start, docker images are often quite large, composed of a multitude of filesystem layers on top of each other. To mitigate the problem a bit, all Cloudron apps share the same base image, that helps a bit. The appsdata folder really only contains the data from within the apps, so that heavily depends on your own usage. The application code, so to speak, is inside those docker images.