disk full - can't access the dashboard

[girish]

docker system df should give you size of the docker images. You can also try cloudron-support --troubleshoot to see if it tells you anything interesting.

Otherwise, you have to du -hcs /home etc to figure where the space is getting used.

[cylon]

Here are the result I have with those command:

root@4v9c:~# du -hcs /home
18G	/home
18G	total
root@4v9c:~# docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          14        14        8.965GB   2.21GB (24%)
Containers      15        14        0B        0B
Local Volumes   46        28        216MB     102MB (47%)
Build Cache     0         0         0B        0B
root@4v9c:~# cloudron-support --troubleshoot
No more space left on /
This is likely the root case of the issue. Free up some space and also check other partitions below:

Filesystem                           Size  Used Avail Use% Mounted on
tmpfs                                392M  2.1M  390M   1% /run
/dev/sda2                             40G   38G     0 100% /
tmpfs                                2.0G     0  2.0G   0% /dev/shm
tmpfs                                5.0M     0  5.0M   0% /run/lock
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/b8a613802b3639564e0604ffb03d10d7ed5fd7a4f1ba7703ee9545290b2a8b2f/merged
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/b1c9c97404110f67157fa968a837d455c89b51ff80c9341096dddd19356775b5/merged
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/b0f743964f5f3183b89865325a08959d53c4b534102c12c461312139e6090253/merged
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/79817dd38f34c049b390e58630e9009fd798b962302df001f23cf4e4620c2380/merged
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/16837309ee1848202e65f0253aa676090e7c892ba840a5b049c0408b43caa2a7/merged
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/b5fe7e94b1a1613ffa654599a1fa91faf4274087817d8ec1df2ad43713869824/merged
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/f65ba77ce0b920234d449ffbf1eccc5ce9fa2e7d24ae57f27c419e87365d4d81/merged
//u378641.your-storagebox.de/backup  1.0T   74G  951G   8% /mnt/volumes/a5fdbaf672ba41589bffb86d6ecd743b
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/a466339b5d66fc034568b89f249b1a980a93ea294158d58747a47f7677ae1d3f/merged
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/068993a6f6f82e4b8b5aa70c155fdf5cd4c8476dc03168e3b39e27dfc35cdc97/merged
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/72a2f687181ba18b78fd90b68bac59ec64e7855c90d25498e2ea5ea9b83eb057/merged
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/ae00c782fcf6eafa8a4bded317aa14da88b036ce0b3890f7928c2c729211b059/merged
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/0cef13e4b6a370c8bc6b65685376bfe26672031de38f9958c5799887b2418784/merged
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/d0335e7fb5c9a922c3ef3f6f55b7199025a0ad51751ed4c7aa07194d0a3f24c4/merged
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/e6ea004d2e793c0f545c58c92194eaec781f38ec15539d4264b7ee15b3297bdd/merged
tmpfs                                392M  4.0K  392M   1% /run/user/0

not sure to understand more. What is this /var/lib/docker/overlay2 ?

Also I see that there is 2GB of reclaimable docker images. Any idea how I can reclaim those ? I tried docker image prune -a but nothing

[timconsidine]

Seems like we need a Disk Full - Recovery FAQ

Awful when it happens
The secret is prevention, especially when disks are small (40Gb is really small)

I encourage everyone to use something, anything , to do monitoring, daily, 12-hourly, 6-hourly, whatever.
Easiest IMHO is a simple bash script on the machine with a cron job to push the results to ntfy

#!/bin/bash
echo 'DeviceName' > /root/ntfy-msg.txt
date >> /root/ntfy-msg.txt
if [ -f /var/run/reboot-required ]; then
 cat /var/run/reboot-required >> /root/ntfy-msg.txt
fi
df -h / >> /root/ntfy-msg.txt
curl -u user:password https://ntfy.domain.tld/device-name -T /root/ntfy-msg.txt

But use anything, just something.
Saves a lot of grief.

[cylon]

Thx for the script will certainly do that once I get back my server. What does the /var/run/reboot-required file contains ? Would be nice if this was included in cloudron out of the box

Any idea what files I can delete to get back some space ?

[timconsidine]

@cylon my script checks for disk free and includes the auto-generated system message about whether the system needs a reboot.
Not important but handy if you’re not regularly logging in to the device (eg all running fine and you’re not changing anything), otherwise you can be unaware that system is asking for it.

[robi]

@timconsidine can we improve that to check the rate of change the disk is being consumed?

If it jumps by several G in an hour unexpectedly, it's a good time to notify.

Rough ex.

#!/bin/bash

initial_state=$(df -H | grep -w / | awk '{print $4}')
sleep 3600
current_state=$(df -H | grep -w / | awk '{print $4}')
rate=$(echo $initial_state - $current_state | bc)
eta=$(echo $current_state / $rate | bc)

echo "Current consumption rate: $current_state rate = $rate /h"
echo "Aprox. time to full disk: $eta"

[timconsidine]

Nice idea
Only tweak I have made in the past is to set a threshold eg 75% full before sending alert

[cylon]

I still can't access my server, anyone knows what file I can delete or how I can reclaim those 2Gb of reclaimable docker images ?

Other option is too increase my server to 80Gb but it would cost me $50 and I still would like to understand what is taking all the space

[girish]

@cylon you can also try docker system prune -a . Also, reclaimable is never perfectly 0 in many cases from what I have seen. No idea what exactly docker is reporting here.

You can try truncating the log files under /home/yellowtent/platformdata/logs to get some space.

[cylon]

docker system prune -a didn't delete anything
There is only 35 Mb in the /home/yellowtent/platformdata/logs

Any idea what is this docker/overlay2 that is taking all the space in the /var/lib folder ?

I'm going to increase to 80Gb...