disk full - can't access the dashboard
-
Hello,
It appears my disk is full and I can't access Cloudron dashboard anymore, nor any other app.
I can ssh into the server through.I would like to clear up some spaces at least to restart the dashboard and check what is taking the space cause I don't store much on the server so this should not happen.
Can you help me regain access to my server ? What files could I safely delete ?
I tried
docker prune image -aas suggested in a post but didn't cleared any spaceFilesystem Size Used Avail Use% Mounted on tmpfs 392M 2.1M 390M 1% /run /dev/sda2 40G 38G 0 100% / tmpfs 2.0G 0 2.0G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock overlay 40G 38G 0 100% /var/lib/docker/overlay2/b8a613802b3639564e0604ffb03d10d7ed5fd7a4f1ba7703ee9545290b2a8b2f/merged overlay 40G 38G 0 100% /var/lib/docker/overlay2/b1c9c97404110f67157fa968a837d455c89b51ff80c9341096dddd19356775b5/merged overlay 40G 38G 0 100% /var/lib/docker/overlay2/b0f743964f5f3183b89865325a08959d53c4b534102c12c461312139e6090253/merged overlay 40G 38G 0 100% /var/lib/docker/overlay2/79817dd38f34c049b390e58630e9009fd798b962302df001f23cf4e4620c2380/merged overlay 40G 38G 0 100% /var/lib/docker/overlay2/16837309ee1848202e65f0253aa676090e7c892ba840a5b049c0408b43caa2a7/merged overlay 40G 38G 0 100% /var/lib/docker/overlay2/b5fe7e94b1a1613ffa654599a1fa91faf4274087817d8ec1df2ad43713869824/merged overlay 40G 38G 0 100% /var/lib/docker/overlay2/f65ba77ce0b920234d449ffbf1eccc5ce9fa2e7d24ae57f27c419e87365d4d81/merged //u378641.your-storagebox.de/backup 1.0T 74G 951G 8% /mnt/volumes/a5fdbaf672ba41589bffb86d6ecd743b overlay 40G 38G 0 100% /var/lib/docker/overlay2/a466339b5d66fc034568b89f249b1a980a93ea294158d58747a47f7677ae1d3f/merged overlay 40G 38G 0 100% /var/lib/docker/overlay2/068993a6f6f82e4b8b5aa70c155fdf5cd4c8476dc03168e3b39e27dfc35cdc97/merged overlay 40G 38G 0 100% /var/lib/docker/overlay2/72a2f687181ba18b78fd90b68bac59ec64e7855c90d25498e2ea5ea9b83eb057/merged overlay 40G 38G 0 100% /var/lib/docker/overlay2/ae00c782fcf6eafa8a4bded317aa14da88b036ce0b3890f7928c2c729211b059/merged overlay 40G 38G 0 100% /var/lib/docker/overlay2/0cef13e4b6a370c8bc6b65685376bfe26672031de38f9958c5799887b2418784/merged overlay 40G 38G 0 100% /var/lib/docker/overlay2/d0335e7fb5c9a922c3ef3f6f55b7199025a0ad51751ed4c7aa07194d0a3f24c4/merged overlay 40G 38G 0 100% /var/lib/docker/overlay2/e6ea004d2e793c0f545c58c92194eaec781f38ec15539d4264b7ee15b3297bdd/merged tmpfs 392M 4.0K 392M 1% /run/user/0 -
Are you using backups on the same disk? This is often the case why disk may fill up (maybe you can carefully delete old backups at
/var/backupsif that is the case)Afterwards to fix the situation we have a troubleshooting guide for that at https://docs.cloudron.io/troubleshooting/#recovery-after-disk-full
-
N nebulon marked this topic as a question on
-
Yes I do, but it seems backups doesn't take much space, only 385Mo. But there is 48Go in
/var/libwhich is strange as I only have 40Go...root@4v9c:~# du -d 1 -h /var 4.0K /var/local 48G /var/lib 385M /var/backups 134M /var/cache 36K /var/tmp 133M /var/log 96K /var/snap 4.0K /var/crash 16K /var/spool 4.0K /var/opt 4.0K /var/mail 12K /var/www 48G /varWhat files or folder can safely be deleted in the
/var/backups?root@4v9c:~# ls -la /var/backups/ total 2100 drwxrwxrwx 14 root root 4096 Feb 1 00:00 . drwxr-xr-x 14 root root 4096 Nov 11 11:13 .. drwxr-xr-x 2 yellowtent yellowtent 4096 Jan 10 23:01 2024-01-10-230139-835 drwxr-xr-x 2 yellowtent yellowtent 4096 Jan 15 23:00 2024-01-15-230042-376 drwxr-xr-x 2 yellowtent yellowtent 4096 Jan 19 23:01 2024-01-19-230103-885 drwxr-xr-x 2 yellowtent yellowtent 4096 Jan 21 23:01 2024-01-21-230152-107 drwxr-xr-x 2 yellowtent yellowtent 4096 Jan 27 23:02 2024-01-27-230000-928 drwxr-xr-x 2 yellowtent yellowtent 4096 Jan 27 23:02 2024-01-27-230203-539 drwxr-xr-x 2 yellowtent yellowtent 4096 Jan 29 23:02 2024-01-29-230000-860 drwxr-xr-x 2 yellowtent yellowtent 4096 Jan 30 03:00 2024-01-30-030006-856 drwxr-xr-x 2 yellowtent yellowtent 4096 Jan 30 05:00 2024-01-30-050001-020 drwxr-xr-x 2 yellowtent yellowtent 4096 Jan 31 01:00 2024-01-31-010003-349 drwxr-xr-x 2 yellowtent yellowtent 4096 Jan 31 03:00 2024-01-31-030000-827 -rw-r--r-- 1 root root 51200 Jan 5 00:00 alternatives.tar.0 -rw-r--r-- 1 root root 2646 Dec 22 00:00 alternatives.tar.1.gz -rw-r--r-- 1 root root 2635 Dec 16 00:00 alternatives.tar.2.gz -rw-r--r-- 1 root root 2636 Dec 12 00:00 alternatives.tar.3.gz -rw-r--r-- 1 root root 2623 Nov 12 00:00 alternatives.tar.4.gz -rw-r--r-- 1 root root 2358 Sep 15 2022 alternatives.tar.5.gz -rw-r--r-- 1 root root 33665 Jan 27 06:08 apt.extended_states.0 -rw-r--r-- 1 root root 3641 Dec 12 06:20 apt.extended_states.1.gz -rw-r--r-- 1 root root 3640 Nov 23 06:48 apt.extended_states.2.gz -rw-r--r-- 1 root root 3643 Nov 11 11:18 apt.extended_states.3.gz -rw-r--r-- 1 root root 3054 Sep 16 2022 apt.extended_states.4.gz -rw-r--r-- 1 root root 3119 Sep 15 2022 apt.extended_states.5.gz -rw-r--r-- 1 root root 3973 Sep 15 2022 apt.extended_states.6.gz -rw-r--r-- 1 root root 0 Feb 1 00:00 dpkg.arch.0 -rw-r--r-- 1 root root 32 Jan 28 00:00 dpkg.arch.1.gz -rw-r--r-- 1 root root 32 Jan 25 00:00 dpkg.arch.2.gz -rw-r--r-- 1 root root 32 Jan 19 00:00 dpkg.arch.3.gz -rw-r--r-- 1 root root 32 Jan 17 00:00 dpkg.arch.4.gz -rw-r--r-- 1 root root 32 Jan 13 00:00 dpkg.arch.5.gz -rw-r--r-- 1 root root 32 Jan 5 00:00 dpkg.arch.6.gz -rw-r--r-- 1 root root 268 Sep 15 2022 dpkg.diversions.0 -rw-r--r-- 1 root root 140 Sep 15 2022 dpkg.diversions.1.gz -rw-r--r-- 1 root root 140 Sep 15 2022 dpkg.diversions.2.gz -rw-r--r-- 1 root root 140 Sep 15 2022 dpkg.diversions.3.gz -rw-r--r-- 1 root root 140 Sep 15 2022 dpkg.diversions.4.gz -rw-r--r-- 1 root root 140 Sep 15 2022 dpkg.diversions.5.gz -rw-r--r-- 1 root root 140 Sep 15 2022 dpkg.diversions.6.gz -rw-r--r-- 1 root root 100 Apr 23 2020 dpkg.statoverride.0 -rw-r--r-- 1 root root 120 Apr 23 2020 dpkg.statoverride.1.gz -rw-r--r-- 1 root root 120 Apr 23 2020 dpkg.statoverride.2.gz -rw-r--r-- 1 root root 120 Apr 23 2020 dpkg.statoverride.3.gz -rw-r--r-- 1 root root 120 Apr 23 2020 dpkg.statoverride.4.gz -rw-r--r-- 1 root root 120 Apr 23 2020 dpkg.statoverride.5.gz -rw-r--r-- 1 root root 120 Apr 23 2020 dpkg.statoverride.6.gz -rw-r--r-- 1 root root 741412 Jan 31 07:00 dpkg.status.0 -rw-r--r-- 1 root root 185249 Jan 27 06:08 dpkg.status.1.gz -rw-r--r-- 1 root root 185145 Jan 24 06:40 dpkg.status.2.gz -rw-r--r-- 1 root root 185144 Jan 18 06:10 dpkg.status.3.gz -rw-r--r-- 1 root root 185145 Jan 16 06:43 dpkg.status.4.gz -rw-r--r-- 1 root root 185146 Jan 12 06:14 dpkg.status.5.gz -rw-r--r-- 1 root root 185141 Jan 4 06:27 dpkg.status.6.gz drwxr-xr-x 2 yellowtent yellowtent 4096 Jan 31 03:00 snapshot -
-
Here are the result I have with those command:
root@4v9c:~# du -hcs /home 18G /home 18G total root@4v9c:~# docker system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 14 14 8.965GB 2.21GB (24%) Containers 15 14 0B 0B Local Volumes 46 28 216MB 102MB (47%) Build Cache 0 0 0B 0B root@4v9c:~# cloudron-support --troubleshoot No more space left on / This is likely the root case of the issue. Free up some space and also check other partitions below: Filesystem Size Used Avail Use% Mounted on tmpfs 392M 2.1M 390M 1% /run /dev/sda2 40G 38G 0 100% / tmpfs 2.0G 0 2.0G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock overlay 40G 38G 0 100% /var/lib/docker/overlay2/b8a613802b3639564e0604ffb03d10d7ed5fd7a4f1ba7703ee9545290b2a8b2f/merged overlay 40G 38G 0 100% /var/lib/docker/overlay2/b1c9c97404110f67157fa968a837d455c89b51ff80c9341096dddd19356775b5/merged overlay 40G 38G 0 100% /var/lib/docker/overlay2/b0f743964f5f3183b89865325a08959d53c4b534102c12c461312139e6090253/merged overlay 40G 38G 0 100% /var/lib/docker/overlay2/79817dd38f34c049b390e58630e9009fd798b962302df001f23cf4e4620c2380/merged overlay 40G 38G 0 100% /var/lib/docker/overlay2/16837309ee1848202e65f0253aa676090e7c892ba840a5b049c0408b43caa2a7/merged overlay 40G 38G 0 100% /var/lib/docker/overlay2/b5fe7e94b1a1613ffa654599a1fa91faf4274087817d8ec1df2ad43713869824/merged overlay 40G 38G 0 100% /var/lib/docker/overlay2/f65ba77ce0b920234d449ffbf1eccc5ce9fa2e7d24ae57f27c419e87365d4d81/merged //u378641.your-storagebox.de/backup 1.0T 74G 951G 8% /mnt/volumes/a5fdbaf672ba41589bffb86d6ecd743b overlay 40G 38G 0 100% /var/lib/docker/overlay2/a466339b5d66fc034568b89f249b1a980a93ea294158d58747a47f7677ae1d3f/merged overlay 40G 38G 0 100% /var/lib/docker/overlay2/068993a6f6f82e4b8b5aa70c155fdf5cd4c8476dc03168e3b39e27dfc35cdc97/merged overlay 40G 38G 0 100% /var/lib/docker/overlay2/72a2f687181ba18b78fd90b68bac59ec64e7855c90d25498e2ea5ea9b83eb057/merged overlay 40G 38G 0 100% /var/lib/docker/overlay2/ae00c782fcf6eafa8a4bded317aa14da88b036ce0b3890f7928c2c729211b059/merged overlay 40G 38G 0 100% /var/lib/docker/overlay2/0cef13e4b6a370c8bc6b65685376bfe26672031de38f9958c5799887b2418784/merged overlay 40G 38G 0 100% /var/lib/docker/overlay2/d0335e7fb5c9a922c3ef3f6f55b7199025a0ad51751ed4c7aa07194d0a3f24c4/merged overlay 40G 38G 0 100% /var/lib/docker/overlay2/e6ea004d2e793c0f545c58c92194eaec781f38ec15539d4264b7ee15b3297bdd/merged tmpfs 392M 4.0K 392M 1% /run/user/0not sure to understand more. What is this
/var/lib/docker/overlay2?Also I see that there is 2GB of reclaimable docker images. Any idea how I can reclaim those ? I tried
docker image prune -abut nothing -
Seems like we need a Disk Full - Recovery FAQ
Awful when it happens
The secret is prevention, especially when disks are small (40Gb is really small)I encourage everyone to use something, anything , to do monitoring, daily, 12-hourly, 6-hourly, whatever.
Easiest IMHO is a simple bash script on the machine with a cron job to push the results tontfy#!/bin/bash echo 'DeviceName' > /root/ntfy-msg.txt date >> /root/ntfy-msg.txt if [ -f /var/run/reboot-required ]; then cat /var/run/reboot-required >> /root/ntfy-msg.txt fi df -h / >> /root/ntfy-msg.txt curl -u user:password https://ntfy.domain.tld/device-name -T /root/ntfy-msg.txtBut use anything, just something.
Saves a lot of grief. -
Thx for the script will certainly do that once I get back my server. What does the
/var/run/reboot-requiredfile contains ? Would be nice if this was included in cloudron out of the box
Any idea what files I can delete to get back some space ?
@cylon my script checks for disk free and includes the auto-generated system message about whether the system needs a reboot.
Not important but handy if you’re not regularly logging in to the device (eg all running fine and you’re not changing anything), otherwise you can be unaware that system is asking for it. -
@cylon my script checks for disk free and includes the auto-generated system message about whether the system needs a reboot.
Not important but handy if you’re not regularly logging in to the device (eg all running fine and you’re not changing anything), otherwise you can be unaware that system is asking for it.@timconsidine can we improve that to check the rate of change the disk is being consumed?
If it jumps by several G in an hour unexpectedly, it's a good time to notify.
Rough ex.
#!/bin/bash initial_state=$(df -H | grep -w / | awk '{print $4}') sleep 3600 current_state=$(df -H | grep -w / | awk '{print $4}') rate=$(echo $initial_state - $current_state | bc) eta=$(echo $current_state / $rate | bc) echo "Current consumption rate: $current_state rate = $rate /h" echo "Aprox. time to full disk: $eta" -
Nice idea
Only tweak I have made in the past is to set a threshold eg 75% full before sending alert -
I still can't access my server, anyone knows what file I can delete or how I can reclaim those 2Gb of reclaimable docker images ?
Other option is too increase my server to 80Gb but it would cost me $50 and I still would like to understand what is taking all the space
-
@cylon you can also try
docker system prune -a. Also, reclaimable is never perfectly 0 in many cases from what I have seen. No idea what exactly docker is reporting here.You can try truncating the log files under /home/yellowtent/platformdata/logs to get some space.
-
G girish has marked this topic as solved on
