"docker-volumes" is filling my whole server storage

[Dont-Worry]

I don't have time to ding into this today. I'll do it tomorrow if i can, but there chances that it is Emby. But there is also other apps that have the same problem. It is not at the same scale because we move more data to Emby but still. I'll keep you informed ASAP.

[Dont-Worry]

Hi @girish and Cloudron team,

I wanted to update you on the situation with the docker-volumes filling up the server storage. After manually cleaning up the Docker volumes, we managed to free up a significant amount of space, but the issue persists as the storage fills up quickly again. Upon investigation, I found that the large files (e.g., mp4, mkv, mp3, wav, srt, webm) I had added to Emby and other applications keep reappearing in the docker-volumes directory.

Here are the details of what I observed:

Volume Details:

• The largest Docker volume taking up space was /var/lib/docker/volumes/6504b9b60123a2ea63aeae015e51ce21d350e8139f743dd0b0c62f1ab5c8e350/_data, which had around 83GB of data.
• I deleted unnecessary files using the following command:

  sudo find /var/lib/docker/volumes/6504b9b60123a2ea63aeae015e51ce21d350e8139f743dd0b0c62f1ab5c8e350/_data -type f \( -name "*.mp4" -o -name "*.mkv" -o -name "*.mp3" -o -name "*.wav" -o -name "*.srt" -o -name "*.webm" \) -exec rm -f {} \;
  

Possible Cause:

• I have a suspicion that this issue might be related to how I added a new storage volume to my server. Here's what I did:
  • Mounted the new storage volume on the server.
  • Added the new storage volume in Cloudron via the admin panel.
• It seems that all the files I add to Emby and possibly other applications using this new volume are also being copied to the base server's storage.

Observation:

• The files accumulating in the docker-volumes are consistently those that were added recently, suggesting a duplication issue. While the new volume works correctly and the files are accessible there, they are also being unnecessarily duplicated on the server's base storage.

Questions for the Cloudron Team:

• Could this be a bug or an issue with how Cloudron handles new storage volumes?
• Is it possible that Cloudron continues to store files on the base server storage even after moving applications to a new volume?
• How can we ensure that files are only stored on the new volume and not duplicated on the server's base storage?

Thank you for your assistance in resolving this issue. Your help and support have always been outstanding, and I hope we can find a solution to this soon. I'm ready to provide any additional information or perform further actions if needed.

[girish]

@Dont-Worry said in "docker-volumes" is filling my whole server storage:

The largest Docker volume taking up space was /var/lib/docker/volumes/6504b9b60123a2ea63aeae015e51ce21d350e8139f743dd0b0c62f1ab5c8e350/_data, which had around 83GB of data.

a) Was this Emby's tmp directory?
b) How are you adding files to Emby. Are you just uploading using the File Manager? Or is this via Emby's upload feature (i.e Emby Premiere)?

[girish]

@Dont-Worry the term volume is used in many contexts and thus can be quite confusing/misleading. The explanation below might be more technical than you want it, but here goes:

• In Docker context, volume refers to storage partitions managed by Docker itself. Docker volumes can be named or anonymous. Cloudron does not use Docker's named volumes. It uses anonymous volumes for /tmp and /run of apps.

• In Cloudron context, volumes are external filesystems that are mounted into app containers. We use docker's "bind mounts" for this.

• In Server context, volumes refer to Block storage. External disks that you can purchase separately from your VPS provider and attach to the server. For example, DO Volume

With this in mind:

• Neither Cloudron nor Docker nor the VPS provider adds things to docker's anonymous volumes (/tmp and /run). Only the app has access to /tmp and /run of an app. These volumes are exclusive to the app and other apps cannot access them (i.e unlike a server, where the /tmp and /run is "shared", in cloudron it is totally separate for each app). This is why I ask if the volume which keeps filling up is Emby's tmp directory. If so, it's either the Emby package or Emby app at fault.

• Neither Cloudron nor Docker nor the VPS provider add things to Cloudron's Volumes . Only the app add/removes stuff here. So again, it's either the Emby package or Emby app at fault.

Hope that helps clarify some confusion. It's pretty much 100% the fault of the app or the app package, but we need to identify the app(s) to fix the root cause.

Also, as a final note: App refers to Emby. App package refers to the Docker image that Cloudron team has built based on Emby binaries and configuration files. The fault could like in one of the two places.

[Dont-Worry]

Hi Girish,

First, thank you for taking the time to provide such a detailed and technical explanation. I genuinely appreciate it, as I find it very insightful.

As I mentioned in an earlier post, the TMP directory that fills up the most is indeed Emby's because we move a lot of data there. We add a substantial amount of content regularly. However, it's not just Emby that has this issue. Several other applications also exhibit similar behavior. The list of these applications, which I posted earlier, includes:

Emby
N8N
Cal
Uptime Kuma
OpenWebUI
Penpot

And a few other unidentified applications
This list is not exhaustive, but these are the primary culprits.

To answer your questions:

a) Yes, the directory that had around 83GB of data was indeed Emby's TMP directory. This directory fills up quickly as we add more content. However, other applications also contribute to the storage issue, just not at the same scale as Emby.

b) I add files to Emby using the File Manager. We do not use Emby's upload feature, even though we have an Emby Premiere instance. We prefer to manage our files directly through Cloudron because we trust its infrastructure and prefer to use it as intended.

I understand that the issue is not inherently with Cloudron itself. My reflection was that this docker-volume issue only started appearing after I added a new storage volume to my server. To clarify, I added an OVH NAS-HA as a new volume to the server, and then I mounted it on Cloudron through the admin panel. This docker-volume did not exist, or at least was not noticeable, before this addition.

Since moving Emby to the new volume, I see my data on the new volume as expected, but it also appears to be duplicated in the TMP directory on the server's disk. This duplication was not happening before the new volume was added. Hence, my hypothesis is that there might be a configuration that wasn't fully adjusted to accommodate the new volume, causing files to be stored in both locations.

I'm not entirely sure how the backend works, so these are just observations and hypotheses based on my use of Cloudron.