disk full - can't access the dashboard

[nebulon]

Are you using backups on the same disk? This is often the case why disk may fill up (maybe you can carefully delete old backups at /var/backups if that is the case)

Afterwards to fix the situation we have a troubleshooting guide for that at https://docs.cloudron.io/troubleshooting/#recovery-after-disk-full

[cylon]

Yes I do, but it seems backups doesn't take much space, only 385Mo. But there is 48Go in /var/lib which is strange as I only have 40Go...

root@4v9c:~# du -d 1 -h  /var
4.0K	/var/local
48G	/var/lib
385M	/var/backups
134M	/var/cache
36K	/var/tmp
133M	/var/log
96K	/var/snap
4.0K	/var/crash
16K	/var/spool
4.0K	/var/opt
4.0K	/var/mail
12K	/var/www
48G	/var

What files or folder can safely be deleted in the /var/backups ?

root@4v9c:~# ls -la /var/backups/
total 2100
drwxrwxrwx 14 root       root         4096 Feb  1 00:00 .
drwxr-xr-x 14 root       root         4096 Nov 11 11:13 ..
drwxr-xr-x  2 yellowtent yellowtent   4096 Jan 10 23:01 2024-01-10-230139-835
drwxr-xr-x  2 yellowtent yellowtent   4096 Jan 15 23:00 2024-01-15-230042-376
drwxr-xr-x  2 yellowtent yellowtent   4096 Jan 19 23:01 2024-01-19-230103-885
drwxr-xr-x  2 yellowtent yellowtent   4096 Jan 21 23:01 2024-01-21-230152-107
drwxr-xr-x  2 yellowtent yellowtent   4096 Jan 27 23:02 2024-01-27-230000-928
drwxr-xr-x  2 yellowtent yellowtent   4096 Jan 27 23:02 2024-01-27-230203-539
drwxr-xr-x  2 yellowtent yellowtent   4096 Jan 29 23:02 2024-01-29-230000-860
drwxr-xr-x  2 yellowtent yellowtent   4096 Jan 30 03:00 2024-01-30-030006-856
drwxr-xr-x  2 yellowtent yellowtent   4096 Jan 30 05:00 2024-01-30-050001-020
drwxr-xr-x  2 yellowtent yellowtent   4096 Jan 31 01:00 2024-01-31-010003-349
drwxr-xr-x  2 yellowtent yellowtent   4096 Jan 31 03:00 2024-01-31-030000-827
-rw-r--r--  1 root       root        51200 Jan  5 00:00 alternatives.tar.0
-rw-r--r--  1 root       root         2646 Dec 22 00:00 alternatives.tar.1.gz
-rw-r--r--  1 root       root         2635 Dec 16 00:00 alternatives.tar.2.gz
-rw-r--r--  1 root       root         2636 Dec 12 00:00 alternatives.tar.3.gz
-rw-r--r--  1 root       root         2623 Nov 12 00:00 alternatives.tar.4.gz
-rw-r--r--  1 root       root         2358 Sep 15  2022 alternatives.tar.5.gz
-rw-r--r--  1 root       root        33665 Jan 27 06:08 apt.extended_states.0
-rw-r--r--  1 root       root         3641 Dec 12 06:20 apt.extended_states.1.gz
-rw-r--r--  1 root       root         3640 Nov 23 06:48 apt.extended_states.2.gz
-rw-r--r--  1 root       root         3643 Nov 11 11:18 apt.extended_states.3.gz
-rw-r--r--  1 root       root         3054 Sep 16  2022 apt.extended_states.4.gz
-rw-r--r--  1 root       root         3119 Sep 15  2022 apt.extended_states.5.gz
-rw-r--r--  1 root       root         3973 Sep 15  2022 apt.extended_states.6.gz
-rw-r--r--  1 root       root            0 Feb  1 00:00 dpkg.arch.0
-rw-r--r--  1 root       root           32 Jan 28 00:00 dpkg.arch.1.gz
-rw-r--r--  1 root       root           32 Jan 25 00:00 dpkg.arch.2.gz
-rw-r--r--  1 root       root           32 Jan 19 00:00 dpkg.arch.3.gz
-rw-r--r--  1 root       root           32 Jan 17 00:00 dpkg.arch.4.gz
-rw-r--r--  1 root       root           32 Jan 13 00:00 dpkg.arch.5.gz
-rw-r--r--  1 root       root           32 Jan  5 00:00 dpkg.arch.6.gz
-rw-r--r--  1 root       root          268 Sep 15  2022 dpkg.diversions.0
-rw-r--r--  1 root       root          140 Sep 15  2022 dpkg.diversions.1.gz
-rw-r--r--  1 root       root          140 Sep 15  2022 dpkg.diversions.2.gz
-rw-r--r--  1 root       root          140 Sep 15  2022 dpkg.diversions.3.gz
-rw-r--r--  1 root       root          140 Sep 15  2022 dpkg.diversions.4.gz
-rw-r--r--  1 root       root          140 Sep 15  2022 dpkg.diversions.5.gz
-rw-r--r--  1 root       root          140 Sep 15  2022 dpkg.diversions.6.gz
-rw-r--r--  1 root       root          100 Apr 23  2020 dpkg.statoverride.0
-rw-r--r--  1 root       root          120 Apr 23  2020 dpkg.statoverride.1.gz
-rw-r--r--  1 root       root          120 Apr 23  2020 dpkg.statoverride.2.gz
-rw-r--r--  1 root       root          120 Apr 23  2020 dpkg.statoverride.3.gz
-rw-r--r--  1 root       root          120 Apr 23  2020 dpkg.statoverride.4.gz
-rw-r--r--  1 root       root          120 Apr 23  2020 dpkg.statoverride.5.gz
-rw-r--r--  1 root       root          120 Apr 23  2020 dpkg.statoverride.6.gz
-rw-r--r--  1 root       root       741412 Jan 31 07:00 dpkg.status.0
-rw-r--r--  1 root       root       185249 Jan 27 06:08 dpkg.status.1.gz
-rw-r--r--  1 root       root       185145 Jan 24 06:40 dpkg.status.2.gz
-rw-r--r--  1 root       root       185144 Jan 18 06:10 dpkg.status.3.gz
-rw-r--r--  1 root       root       185145 Jan 16 06:43 dpkg.status.4.gz
-rw-r--r--  1 root       root       185146 Jan 12 06:14 dpkg.status.5.gz
-rw-r--r--  1 root       root       185141 Jan  4 06:27 dpkg.status.6.gz
drwxr-xr-x  2 yellowtent yellowtent   4096 Jan 31 03:00 snapshot

[girish]

docker system df should give you size of the docker images. You can also try cloudron-support --troubleshoot to see if it tells you anything interesting.

Otherwise, you have to du -hcs /home etc to figure where the space is getting used.

[cylon]

Here are the result I have with those command:

root@4v9c:~# du -hcs /home
18G	/home
18G	total
root@4v9c:~# docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          14        14        8.965GB   2.21GB (24%)
Containers      15        14        0B        0B
Local Volumes   46        28        216MB     102MB (47%)
Build Cache     0         0         0B        0B
root@4v9c:~# cloudron-support --troubleshoot
No more space left on /
This is likely the root case of the issue. Free up some space and also check other partitions below:

Filesystem                           Size  Used Avail Use% Mounted on
tmpfs                                392M  2.1M  390M   1% /run
/dev/sda2                             40G   38G     0 100% /
tmpfs                                2.0G     0  2.0G   0% /dev/shm
tmpfs                                5.0M     0  5.0M   0% /run/lock
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/b8a613802b3639564e0604ffb03d10d7ed5fd7a4f1ba7703ee9545290b2a8b2f/merged
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/b1c9c97404110f67157fa968a837d455c89b51ff80c9341096dddd19356775b5/merged
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/b0f743964f5f3183b89865325a08959d53c4b534102c12c461312139e6090253/merged
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/79817dd38f34c049b390e58630e9009fd798b962302df001f23cf4e4620c2380/merged
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/16837309ee1848202e65f0253aa676090e7c892ba840a5b049c0408b43caa2a7/merged
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/b5fe7e94b1a1613ffa654599a1fa91faf4274087817d8ec1df2ad43713869824/merged
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/f65ba77ce0b920234d449ffbf1eccc5ce9fa2e7d24ae57f27c419e87365d4d81/merged
//u378641.your-storagebox.de/backup  1.0T   74G  951G   8% /mnt/volumes/a5fdbaf672ba41589bffb86d6ecd743b
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/a466339b5d66fc034568b89f249b1a980a93ea294158d58747a47f7677ae1d3f/merged
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/068993a6f6f82e4b8b5aa70c155fdf5cd4c8476dc03168e3b39e27dfc35cdc97/merged
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/72a2f687181ba18b78fd90b68bac59ec64e7855c90d25498e2ea5ea9b83eb057/merged
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/ae00c782fcf6eafa8a4bded317aa14da88b036ce0b3890f7928c2c729211b059/merged
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/0cef13e4b6a370c8bc6b65685376bfe26672031de38f9958c5799887b2418784/merged
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/d0335e7fb5c9a922c3ef3f6f55b7199025a0ad51751ed4c7aa07194d0a3f24c4/merged
overlay                               40G   38G     0 100% /var/lib/docker/overlay2/e6ea004d2e793c0f545c58c92194eaec781f38ec15539d4264b7ee15b3297bdd/merged
tmpfs                                392M  4.0K  392M   1% /run/user/0

not sure to understand more. What is this /var/lib/docker/overlay2 ?

Also I see that there is 2GB of reclaimable docker images. Any idea how I can reclaim those ? I tried docker image prune -a but nothing

[timconsidine]

Seems like we need a Disk Full - Recovery FAQ

Awful when it happens
The secret is prevention, especially when disks are small (40Gb is really small)

I encourage everyone to use something, anything , to do monitoring, daily, 12-hourly, 6-hourly, whatever.
Easiest IMHO is a simple bash script on the machine with a cron job to push the results to ntfy

#!/bin/bash
echo 'DeviceName' > /root/ntfy-msg.txt
date >> /root/ntfy-msg.txt
if [ -f /var/run/reboot-required ]; then
 cat /var/run/reboot-required >> /root/ntfy-msg.txt
fi
df -h / >> /root/ntfy-msg.txt
curl -u user:password https://ntfy.domain.tld/device-name -T /root/ntfy-msg.txt

But use anything, just something.
Saves a lot of grief.

[cylon]

Thx for the script will certainly do that once I get back my server. What does the /var/run/reboot-required file contains ? Would be nice if this was included in cloudron out of the box

Any idea what files I can delete to get back some space ?

[timconsidine]

@cylon my script checks for disk free and includes the auto-generated system message about whether the system needs a reboot.
Not important but handy if you’re not regularly logging in to the device (eg all running fine and you’re not changing anything), otherwise you can be unaware that system is asking for it.

[robi]

@timconsidine can we improve that to check the rate of change the disk is being consumed?

If it jumps by several G in an hour unexpectedly, it's a good time to notify.

Rough ex.

#!/bin/bash

initial_state=$(df -H | grep -w / | awk '{print $4}')
sleep 3600
current_state=$(df -H | grep -w / | awk '{print $4}')
rate=$(echo $initial_state - $current_state | bc)
eta=$(echo $current_state / $rate | bc)

echo "Current consumption rate: $current_state rate = $rate /h"
echo "Aprox. time to full disk: $eta"

[timconsidine]

Nice idea
Only tweak I have made in the past is to set a threshold eg 75% full before sending alert

[cylon]

I still can't access my server, anyone knows what file I can delete or how I can reclaim those 2Gb of reclaimable docker images ?

Other option is too increase my server to 80Gb but it would cost me $50 and I still would like to understand what is taking all the space

[girish]

@cylon you can also try docker system prune -a . Also, reclaimable is never perfectly 0 in many cases from what I have seen. No idea what exactly docker is reporting here.

You can try truncating the log files under /home/yellowtent/platformdata/logs to get some space.

[cylon]

docker system prune -a didn't delete anything
There is only 35 Mb in the /home/yellowtent/platformdata/logs

Any idea what is this docker/overlay2 that is taking all the space in the /var/lib folder ?

I'm going to increase to 80Gb...