"docker-volumes" is filling my whole server storage

[Dont-Worry]

I'm currently experiencing an issue with my Cloudron server where the /tmp directories in several Docker containers are consuming a significant amount of disk space. Following the initial guidance, I performed the following steps:

1. Checked the mounted volumes for each container:
   Using the command sudo docker ps -a --format '{{ .ID }}' | xargs -I {} sudo docker inspect -f '{{ .ID }} {{ .Name }}{{ printf "\n" }}{{ range .Mounts }}{{ printf "\n\t" }}{{ .Type }} {{ if eq .Type "bind" }}{{ .Source }}{{ end }}{{ .Name }} => {{ .Destination }}{{ end }}{{ printf "\n" }}' {}, I identified the volumes mounted in each container and found that several containers have significant data in their /tmp directories.

2. Detailed inspection of /tmp directories:
   I listed the contents of the /tmp directories in each container using:

   containers=$(sudo docker ps -a -q)
   for container in $containers; do
     echo "Checking /tmp in container $container:"
     sudo docker exec -it $container sh -c "du -sh /tmp/*" || echo "/tmp is empty or not accessible"
     echo "-----------------------"
   done
   

   This revealed that several containers have large files or directories within /tmp, some reaching up to several gigabytes.

3. Plan to clean /tmp directories:
   Based on the findings, I plan to clean up the /tmp directories using the following command:

   containers=$(sudo docker ps -a -q)
   for container in $containers; do
     echo "Cleaning /tmp in container $container:"
     sudo docker exec -it $container sh -c "rm -rf /tmp/*" || echo "/tmp is empty or not accessible"
     echo "-----------------------"
   done
   

However, before proceeding, I wanted to consult with the Cloudron team to ensure that this approach won't inadvertently affect any critical application functionality.

Additionally, here are the details regarding the container consuming the most space:

Inside one of the containers, it seems that there are parts of Emby content (videos and audio files) that should not be there because Emby was moved to a new volume. However, the server disk still has these files:

user:~$ sudo docker exec -it docker_ID /bin/bash
root@sftp:/app/code# cd /tmp
root@sftp:/tmp# ls -lah
total 130G
-rw-r--r-- 1 root root  631M May 20 12:43 -9Xo-89Hb6E0UzpXKPvMtvT_.mp4
-rw-r--r-- 1 root root  2.5G May 20 16:13 -B0LYra-w572_N_Fl-z1Fg8Y.mp4
-rw-r--r-- 1 root root  971M May 20 13:54 -H0OPKJKqQmK53HFZTtxHkCe.mp4
-rw-r--r-- 1 root root  727M May 20 17:56 -XLVWBgNrI4b4bLPiShFvnAo.mp4
-rw-r--r-- 1 root root  1.7G May 20 17:52 -kzTrAAk7QbN6242Zsh0y2WG.mp4
-rw-r--r-- 1 root root   38M May 29 11:20 0AO3hMZZHlD0vqoaeWODHD3u.wav
-rw-r--r-- 1 root root  1.9G May 20 17:22 0K4x3d-JnMca3po-P3XxyQVa.mp4
-rw-r--r-- 1 root root  773M May 30 20:49 0PD5NHwvcHCyqBW10OHjXNp6.mp4
-rw-r--r-- 1 root root   32M May 29 10:57 0gVIwfG9iUrr4LgZ7B4nj0lh.wav
-rw-r--r-- 1 root root  285M May 20 12:51 0ivU3H-H5UVFr_pGHnUZC5i8.mp4
...
-rw-r--r-- 1 root root  1.2G May 20 13:54 YSJU8K5Caudu1oMhX0-a5HIS.mp4
-rw-r--r-- 1 root root  3.6G May 20 19:36 YW64K1HL_7oyQEwJCfbz5nKH.mp4
-rw-r--r-- 1 root root  270M May 26 11:56 YiSQ7xKap3kpTiJPwFaUipI2.mp4
-rw-r--r-- 1 root root  190M May 20 20:31 YqP5gd4str1rdX4Fxa1_X7uz.mp4
-rw-r--r-- 1 root root  2.2G May 20 14:23 ZCtXoF9dzT5KjJxw0QOnhh8Z.mp4
...

Questions:

1. Is there a recommended way to handle large /tmp directories in Docker containers on a Cloudron server?
2. Are there any specific precautions I should take before cleaning these directories?
3. Is there a way to prevent these large files from accumulating in /tmp in the future?

Any guidance or recommendations would be greatly appreciated.

[girish]

@Dont-Worry Not sure why all those files are there. Those files are not created by Cloudron itself, we don't use an app's tmp directory. Maybe some Emby feature is buggy? I use Emby as well for many years now and atleast my Emby's tmp here is practically empty.

Cloudron has some logic to cleanup tmp of apps. It deletes 10 days old files using find /tmp -type f -mtime +10 -exec rm -rf {} + .

[girish]

@Dont-Worry said in "docker-volumes" is filling my whole server storage:

Cloudron server where the /tmp directories in several Docker containers are consuming a significant amount of disk space

I thought the issue is Emby primarily. Do you see this for other apps too?

[girish]

@Dont-Worry said in "docker-volumes" is filling my whole server storage:

Any guidance or recommendations would be greatly appreciated.

I would just cleanup Emby's tmp by hand now (just rm then). After that, periodically monitor the tmp to see what is getting accumulated there.

[Dont-Worry]

Thank you for your guidance. Following your recommendations, we have successfully managed to free up a significant amount of storage. Here's an overview of what we did:

Identified the issue: We found that a large number of video and audio files were accumulated in the /tmp directory of one of the Docker containers (even we still don't understand how they got there). These files were not expected to be there as we had already moved Emby to a new volume.

Cleaned up manually: As you suggested, we manually cleaned up the /tmp directory of the affected container. This involved deleting all the unnecessary files that were consuming a lot of space.

Before the cleanup, the Docker volumes were consuming approximately 158GB of storage. After the cleanup, we managed to reduce it down to just 15GB.

Here's a brief look at the cleanup process:

user:~$ sudo docker exec -it <container_id> /bin/bash
root@sftp:/app/code# cd /tmp
root@sftp:/tmp# ls -lah
...

# Deleted all the large unwanted files
root@sftp:/tmp# rm -rf *.mp4 *.wav

We will continue to monitor the /tmp directories in our Docker containers to ensure they do not fill up again. For now, this manual cleanup has significantly resolved our storage issue.

[Dont-Worry]

@girish there is still 15gb of data in the docker-volumes, I hope this is not gonna do the same. But for now this solution is working for short term use.

[girish]

@Dont-Worry When you have time, dig into those 15GB too. Maybe some /run or /tmp is getting filled up with something. We have to do this app by app to understand why it's getting filled up. Because as mentioned, Cloudron doesn't access those directories, it's part of the apps.

[Dont-Worry]

Thanks for the guidance @girish . We've followed your recommendations and conducted a thorough cleanup of the /tmp and /run directories within our Docker containers. Here’s a summary of our findings and actions taken:

Containers Cleaned:

Emby
N8N
Cal
Uptime Kuma
OpenWebUI
Penpot
A few other containers which I could not identify precisely

Actions Taken:

Cleared substantial temporary and cache files from the /run and /tmp directories across these containers.
Specifically, we found large amounts of data in the /run directories of Calcom and Penpot, and in the /tmp directories of Penpot and N8N.

Results:

We successfully freed up approximately 8GB of space.
There are still about 8.5GB left in the docker-volumes.
Despite the cleanup, it appears that some applications, particularly the ones mentioned, might still have some lingering files contributing to the remaining usage. We will continue to monitor the disk usage and investigate further if necessary.

[Dont-Worry]

Hi,

I wanted to provide an update on the ongoing issue with my server storage being filled by docker-volumes. Despite performing manual cleanups and managing Docker volumes, the problem persists. The data continues to reappear, causing storage usage to climb rapidly.

A couple of days ago, after extensive cleaning, we managed to free up space and bring it down to around 20GB. However, the storage usage has been increasing at an alarming rate of approximately 20GB per day. This rapid growth is not only concerning but also quite unstable, making it difficult to maintain a functional server.

Having to manually clean up the storage repeatedly is both time-consuming and frustrating. This situation is unsustainable in the long term, and I am reaching out to see if anyone has further insights into why this is happening. What exactly is causing the docker-volumes to fill up so quickly? Is there a specific application or a known issue that might be contributing to this problem?

Any advice or solutions to address this issue more permanently would be greatly appreciated.

There is 20gb that has been added between this morning 7AM (63GB in Docker-Volumes) and now (80.64GB).

[girish]

@Dont-Worry said in "docker-volumes" is filling my whole server storage:

What exactly is causing the docker-volumes to fill up so quickly? Is there a specific application or a known issue that might be contributing to this problem?

Have you identified which apps are filling up the volumes this time around? Is this emby again?

[Dont-Worry]

I don't have time to ding into this today. I'll do it tomorrow if i can, but there chances that it is Emby. But there is also other apps that have the same problem. It is not at the same scale because we move more data to Emby but still. I'll keep you informed ASAP.

[Dont-Worry]

Hi @girish and Cloudron team,

I wanted to update you on the situation with the docker-volumes filling up the server storage. After manually cleaning up the Docker volumes, we managed to free up a significant amount of space, but the issue persists as the storage fills up quickly again. Upon investigation, I found that the large files (e.g., mp4, mkv, mp3, wav, srt, webm) I had added to Emby and other applications keep reappearing in the docker-volumes directory.

Here are the details of what I observed:

Volume Details:

• The largest Docker volume taking up space was /var/lib/docker/volumes/6504b9b60123a2ea63aeae015e51ce21d350e8139f743dd0b0c62f1ab5c8e350/_data, which had around 83GB of data.
• I deleted unnecessary files using the following command:

  sudo find /var/lib/docker/volumes/6504b9b60123a2ea63aeae015e51ce21d350e8139f743dd0b0c62f1ab5c8e350/_data -type f \( -name "*.mp4" -o -name "*.mkv" -o -name "*.mp3" -o -name "*.wav" -o -name "*.srt" -o -name "*.webm" \) -exec rm -f {} \;
  

Possible Cause:

• I have a suspicion that this issue might be related to how I added a new storage volume to my server. Here's what I did:
  • Mounted the new storage volume on the server.
  • Added the new storage volume in Cloudron via the admin panel.
• It seems that all the files I add to Emby and possibly other applications using this new volume are also being copied to the base server's storage.

Observation:

• The files accumulating in the docker-volumes are consistently those that were added recently, suggesting a duplication issue. While the new volume works correctly and the files are accessible there, they are also being unnecessarily duplicated on the server's base storage.

Questions for the Cloudron Team:

• Could this be a bug or an issue with how Cloudron handles new storage volumes?
• Is it possible that Cloudron continues to store files on the base server storage even after moving applications to a new volume?
• How can we ensure that files are only stored on the new volume and not duplicated on the server's base storage?

Thank you for your assistance in resolving this issue. Your help and support have always been outstanding, and I hope we can find a solution to this soon. I'm ready to provide any additional information or perform further actions if needed.

[girish]

@Dont-Worry said in "docker-volumes" is filling my whole server storage:

The largest Docker volume taking up space was /var/lib/docker/volumes/6504b9b60123a2ea63aeae015e51ce21d350e8139f743dd0b0c62f1ab5c8e350/_data, which had around 83GB of data.

a) Was this Emby's tmp directory?
b) How are you adding files to Emby. Are you just uploading using the File Manager? Or is this via Emby's upload feature (i.e Emby Premiere)?

[girish]

@Dont-Worry the term volume is used in many contexts and thus can be quite confusing/misleading. The explanation below might be more technical than you want it, but here goes:

• In Docker context, volume refers to storage partitions managed by Docker itself. Docker volumes can be named or anonymous. Cloudron does not use Docker's named volumes. It uses anonymous volumes for /tmp and /run of apps.

• In Cloudron context, volumes are external filesystems that are mounted into app containers. We use docker's "bind mounts" for this.

• In Server context, volumes refer to Block storage. External disks that you can purchase separately from your VPS provider and attach to the server. For example, DO Volume

With this in mind:

• Neither Cloudron nor Docker nor the VPS provider adds things to docker's anonymous volumes (/tmp and /run). Only the app has access to /tmp and /run of an app. These volumes are exclusive to the app and other apps cannot access them (i.e unlike a server, where the /tmp and /run is "shared", in cloudron it is totally separate for each app). This is why I ask if the volume which keeps filling up is Emby's tmp directory. If so, it's either the Emby package or Emby app at fault.

• Neither Cloudron nor Docker nor the VPS provider add things to Cloudron's Volumes . Only the app add/removes stuff here. So again, it's either the Emby package or Emby app at fault.

Hope that helps clarify some confusion. It's pretty much 100% the fault of the app or the app package, but we need to identify the app(s) to fix the root cause.

Also, as a final note: App refers to Emby. App package refers to the Docker image that Cloudron team has built based on Emby binaries and configuration files. The fault could like in one of the two places.

[Dont-Worry]

Hi Girish,

First, thank you for taking the time to provide such a detailed and technical explanation. I genuinely appreciate it, as I find it very insightful.

As I mentioned in an earlier post, the TMP directory that fills up the most is indeed Emby's because we move a lot of data there. We add a substantial amount of content regularly. However, it's not just Emby that has this issue. Several other applications also exhibit similar behavior. The list of these applications, which I posted earlier, includes:

Emby
N8N
Cal
Uptime Kuma
OpenWebUI
Penpot

And a few other unidentified applications
This list is not exhaustive, but these are the primary culprits.

To answer your questions:

a) Yes, the directory that had around 83GB of data was indeed Emby's TMP directory. This directory fills up quickly as we add more content. However, other applications also contribute to the storage issue, just not at the same scale as Emby.

b) I add files to Emby using the File Manager. We do not use Emby's upload feature, even though we have an Emby Premiere instance. We prefer to manage our files directly through Cloudron because we trust its infrastructure and prefer to use it as intended.

I understand that the issue is not inherently with Cloudron itself. My reflection was that this docker-volume issue only started appearing after I added a new storage volume to my server. To clarify, I added an OVH NAS-HA as a new volume to the server, and then I mounted it on Cloudron through the admin panel. This docker-volume did not exist, or at least was not noticeable, before this addition.

Since moving Emby to the new volume, I see my data on the new volume as expected, but it also appears to be duplicated in the TMP directory on the server's disk. This duplication was not happening before the new volume was added. Hence, my hypothesis is that there might be a configuration that wasn't fully adjusted to accommodate the new volume, causing files to be stored in both locations.

I'm not entirely sure how the backend works, so these are just observations and hypotheses based on my use of Cloudron.