"Platform data" and "Other" disk usage

[girish]

The prune commands should be safe to run. Unless there is some Cloudron bug, it shouldn't remove anything.

[jk]

@girish thank you.

That worked for docker volume prune. The other command (docker image prune) did not actually clean anything up though, but the output of docker system df stays with 66% reclaimable.

Any further ideas?

[yusf]

@girish said in "Platform data" and "Other" disk usage:

We have a logrotate issue which is not cleaning up logs properly.

I still need to continuously watch these log files not filling up the machine completely. Can you take another look at the log rotation feature?

[girish]

@yusf Do you know which log is taking up space? Maybe we miss a logrotation rule somewhere. You can find this by du -hcs /home/yellowtent/platformdata/logs/*

[d19dotca]

@girish For what it's worth, I ran that command and found the total size was 872 MB. Is that expected? It seems like a lot to me to be nearly a GB in size just for text files. I'd expect that to be more around the 200 MB mark or so, but I'm not sure of the logrotate rules and how much gets logged generally nor how quickly.

I noticed one of the larger log files was around 47 MB on its own, and it looks to be from my Radicale app. It seems to be logging in DEBUG mode, which I presume is not intended? I'm guessing for production systems we'd like to keep logs in a normal INFO mode, right?

[girish]

mm, OK. I looked into your own installation - https://paste.cloudron.io/eqixarerox.css (~800MB)

logrotate configs are in /home/yellowtent/platformdata/logrotate.d

• Some apps like 2f004a04-e256-426b-9485-e06a31adc2dd were missing configs. This is because when we added the new configs many releases ago, we didn't go back and re-generate it for all the apps. To fix this, simply enable and disable recovery mode and then the logrotate config will be generated.

• For turn, the logrotate was not added when we added it in the previous release. I have fixed this now for next release.

• For all apps/services, we keep up to 2 logs of 10M. This is why most of them are < 20M. So, this is mostly working as expected.

• For the rest, I see that log files are > 20M. I wonder why... investigating.

[girish]

Found out why. If a log file is > 20M at rotation time, then it's never truncated. Looks like this is just how logrotate works.

[necrevistonnezr]

https://paste.cloudron.io/upozogutut.sql
For me it adds up to 699 MB

Is there a way to clean up those logs in between?

[girish]

@necrevistonnezr If you go into /home/yellowtent/platformdata/logrotate, do you see 07e4e251-ed90-42ca-9ba3-e015b108f1e6 and c5c137fa-0221-4d42-b418-10e544cebad1 there? If not, go those apps (you can figure this out either by looking at the logs a bit or in the dashboard's URL) and start recovery mode and end recovery mode. This will generate the logrotate configs for them.

Once, the configs are in place, future logs will get rotated. But this does not help the current logs. You can just truncate them like:

truncate -s0 /home/yellowtent/platformdata/logs/07e4e251-ed90-42ca-9ba3-e015b108f1e6/app.log

[necrevistonnezr]

@girish Thanks, worked!

[foliovision]

Hi Girish,

We've run into issues with reclaimable space as well. Here's where we stand now.

# docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          18        18        15.99GB   4.727GB (29%)
Containers      19        19        95.79kB   0B (0%)
Local Volumes   62        58        623.4MB   3.582kB (0%)
Build Cache     0         0         0B        0B

This is after running docker image prune -a which removed 19 Images (saved 4 GB), 3 Containers, 324 Local Volumes (saved 2.2 GB). We have also already ran We tried docker volume prune and docker system prune too. These only saved about 100 MB each.

How do we reclaim the last 4.727GB of reclaimable space?

Thanks in advance for your good advice, Alec

PS. I've posted on this older thread as it's also about reclaimable space, to avoid too many duplicate threads on the forum.

[girish]

@foliovision you can also try docker system prune -a --volumes but generally I have never succeeded in making that reclaimable space be less than 2GB in any server using docker (even without Cloudron).