Disk full after update
-
My box was prompting me to reboot after the latest package updates and now refuses to resurface because it's out of disk space
According to my research, the disk space used should be 46G out of 75G but df -h shows 75G used by the overlay file system(s):
root@my:/# du -ch -t 1024m 15G ./home/yellowtent/boxdata/mail/vmail/****/mail 15G ./home/yellowtent/boxdata/mail/vmail/**** 15G ./home/yellowtent/boxdata/mail/vmail 15G ./home/yellowtent/boxdata/mail 15G ./home/yellowtent/boxdata 1.7G ./home/yellowtent/platformdata/mysql/2e6eb3bd9d3a9626 2.0G ./home/yellowtent/platformdata/mysql 1.2G ./home/yellowtent/platformdata/logs 4.0G ./home/yellowtent/platformdata 1.7G ./home/yellowtent/appsdata 21G ./home/yellowtent 21G ./home 1.3G ./var/lib/docker/volumes/eb0fd2ce95a38087ad725cf43ba1479e74bfa90f1ca29be2cecdb5d212e7a00b/_data/collaboraoffice 1.4G ./var/lib/docker/volumes/eb0fd2ce95a38087ad725cf43ba1479e74bfa90f1ca29be2cecdb5d212e7a00b/_data 1.4G ./var/lib/docker/volumes/eb0fd2ce95a38087ad725cf43ba1479e74bfa90f1ca29be2cecdb5d212e7a00b 1.3G ./var/lib/docker/volumes/43e615fdc4e362f869254b9c53fd70b5db008d7cc6dcc39d450735d77851ad46/_data/collaboraoffice 1.4G ./var/lib/docker/volumes/43e615fdc4e362f869254b9c53fd70b5db008d7cc6dcc39d450735d77851ad46/_data 1.4G ./var/lib/docker/volumes/43e615fdc4e362f869254b9c53fd70b5db008d7cc6dcc39d450735d77851ad46 3.7G ./var/lib/docker/volumes 1.1G ./var/lib/docker/overlay2/06619deef5372d52ad33f368ae13989db4af9d88c284fb2bdd3a7cd22f64fdaf/merged/usr/lib 2.2G ./var/lib/docker/overlay2/06619deef5372d52ad33f368ae13989db4af9d88c284fb2bdd3a7cd22f64fdaf/merged/usr 2.6G ./var/lib/docker/overlay2/06619deef5372d52ad33f368ae13989db4af9d88c284fb2bdd3a7cd22f64fdaf/merged 2.6G ./var/lib/docker/overlay2/06619deef5372d52ad33f368ae13989db4af9d88c284fb2bdd3a7cd22f64fdaf 1.1G ./var/lib/docker/overlay2/10455ab22a1a211573f5caaf95ed2bb087447252b65489b21ff61ef45d145351/merged/usr/lib 2.2G ./var/lib/docker/overlay2/10455ab22a1a211573f5caaf95ed2bb087447252b65489b21ff61ef45d145351/merged/usr 2.3G ./var/lib/docker/overlay2/10455ab22a1a211573f5caaf95ed2bb087447252b65489b21ff61ef45d145351/merged 2.3G ./var/lib/docker/overlay2/10455ab22a1a211573f5caaf95ed2bb087447252b65489b21ff61ef45d145351 1.9G ./var/lib/docker/overlay2/7cc0ed857f244984e86fb2ba98a894a14512fe77bc702465adba6a3ca7fb8487/diff/app/code/packages 2.6G ./var/lib/docker/overlay2/7cc0ed857f244984e86fb2ba98a894a14512fe77bc702465adba6a3ca7fb8487/diff/app/code 2.6G ./var/lib/docker/overlay2/7cc0ed857f244984e86fb2ba98a894a14512fe77bc702465adba6a3ca7fb8487/diff/app 2.9G ./var/lib/docker/overlay2/7cc0ed857f244984e86fb2ba98a894a14512fe77bc702465adba6a3ca7fb8487/diff 2.9G ./var/lib/docker/overlay2/7cc0ed857f244984e86fb2ba98a894a14512fe77bc702465adba6a3ca7fb8487 1.1G ./var/lib/docker/overlay2/6d96c852b5d2b8a33e76cccc6f9a4727fa46a982fc567e34d87cde34949616be/diff/usr/lib 1.6G ./var/lib/docker/overlay2/6d96c852b5d2b8a33e76cccc6f9a4727fa46a982fc567e34d87cde34949616be/diff/usr 1.7G ./var/lib/docker/overlay2/6d96c852b5d2b8a33e76cccc6f9a4727fa46a982fc567e34d87cde34949616be/diff 1.7G ./var/lib/docker/overlay2/6d96c852b5d2b8a33e76cccc6f9a4727fa46a982fc567e34d87cde34949616be 1.4G ./var/lib/docker/overlay2/95628946ec04929aae79fe8046bd20af7c801a07e7b45ce8101e065be9bebb71/merged/usr/local/share/.cache/yarn/v6 1.4G ./var/lib/docker/overlay2/95628946ec04929aae79fe8046bd20af7c801a07e7b45ce8101e065be9bebb71/merged/usr/local/share/.cache/yarn 1.4G ./var/lib/docker/overlay2/95628946ec04929aae79fe8046bd20af7c801a07e7b45ce8101e065be9bebb71/merged/usr/local/share/.cache 1.4G ./var/lib/docker/overlay2/95628946ec04929aae79fe8046bd20af7c801a07e7b45ce8101e065be9bebb71/merged/usr/local/share 1.9G ./var/lib/docker/overlay2/95628946ec04929aae79fe8046bd20af7c801a07e7b45ce8101e065be9bebb71/merged/usr/local 1.1G ./var/lib/docker/overlay2/95628946ec04929aae79fe8046bd20af7c801a07e7b45ce8101e065be9bebb71/merged/usr/lib 3.6G ./var/lib/docker/overlay2/95628946ec04929aae79fe8046bd20af7c801a07e7b45ce8101e065be9bebb71/merged/usr 4.6G ./var/lib/docker/overlay2/95628946ec04929aae79fe8046bd20af7c801a07e7b45ce8101e065be9bebb71/merged 4.6G ./var/lib/docker/overlay2/95628946ec04929aae79fe8046bd20af7c801a07e7b45ce8101e065be9bebb71 1.1G ./var/lib/docker/overlay2/cd8242f93484524500811c52c3db963afce7af67e1402c1d0009c597878910d7/merged/usr/lib 2.2G ./var/lib/docker/overlay2/cd8242f93484524500811c52c3db963afce7af67e1402c1d0009c597878910d7/merged/usr du: fts_read failed: ./var/lib/docker/overlay2/cd8242f93484524500811c52c3db963afce7af67e1402c1d0009c597878910d7/merged: No such file or directory 46G totalyet
root@my:/# df -h Filesystem Size Used Avail Use% Mounted on udev 1.9G 0 1.9G 0% /dev tmpfs 384M 2.5M 382M 1% /run /dev/sda1 75G 74G 0 100% / tmpfs 1.9G 0 1.9G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup /dev/sda15 253M 5.2M 247M 3% /boot/efi overlay 75G 74G 0 100% /var/lib/docker/overlay2/06619deef5372d52ad33f368ae13989db4af9d88c284fb2bdd3a7cd22f64fdaf/merged overlay 75G 74G 0 100% /var/lib/docker/overlay2/f4c82f7c6f800a006af5c1f3f0c9e5a04d2e91ea1116df30f82b03fb72d5f4ac/merged overlay 75G 74G 0 100% /var/lib/docker/overlay2/10455ab22a1a211573f5caaf95ed2bb087447252b65489b21ff61ef45d145351/merged overlay 75G 74G 0 100% /var/lib/docker/overlay2/f39da0d113262407c436b4f8a5952c94bce32aa550eff18bdb6791df757fb259/merged overlay 75G 74G 0 100% /var/lib/docker/overlay2/6f7dd5920dd9595ef626d0531ccd8eb57bc8fa1d09ff3c9ba7bdab354a6a834a/merged overlay 75G 74G 0 100% /var/lib/docker/overlay2/c77865d8d5bf4f7149f0a20a2ea4078e9ebe5aa7ae539daa8b1c5790272f5089/merged overlay 75G 74G 0 100% /var/lib/docker/overlay2/e5e74cd6999b21caca6632a4b2e1be0b0c1c5665d9a77e25c97d223106ca9e9f/merged overlay 75G 74G 0 100% /var/lib/docker/overlay2/7d7e953ff32324575fd42f9bed6f41f1413a68e78a109dbe3fee57aaefbae05e/merged overlay 75G 74G 0 100% /var/lib/docker/overlay2/d31fc02ad295d6045e24fbf1349df485d6c6616fb48ffa6a9b440af3bc772ed2/merged overlay 75G 74G 0 100% /var/lib/docker/overlay2/9ed98d68929478b7c5617fb912de54b028d0cb9c4026882c5e62af7a9dc6f1b7/merged overlay 75G 74G 0 100% /var/lib/docker/overlay2/761d1029ed47fe60c26b91c30d620d213c135e22dad486e75c1aeb13fa12bf96/merged overlay 75G 74G 0 100% /var/lib/docker/overlay2/1afa4fe659b317a5a004f2d6871df3b1b11af62ac22bdb495a9492e5fd42a4bf/merged overlay 75G 74G 0 100% /var/lib/docker/overlay2/d21bcb497a014767b3fa030bb295467f90e46d4d45907482b82c4c1eb5ffa33d/merged overlay 75G 74G 0 100% /var/lib/docker/overlay2/7fade2e1d24f73d71c01a8aceb16e1a144f6419b11272f4618ba7394622ef2bb/merged overlay 75G 74G 0 100% /var/lib/docker/overlay2/09b7f60915e418a06e19310c28a0161444f486d86f977fa822b34d11315b2735/merged overlay 75G 74G 0 100% /var/lib/docker/overlay2/e52d8bfa8c01be3c61b64a880a8a35222a853ed28e62f392d9e7b7e5f4cac568/merged tmpfs 384M 0 384M 0% /run/user/0 overlay 75G 74G 0 100% /var/lib/docker/overlay2/cd8242f93484524500811c52c3db963afce7af67e1402c1d0009c597878910d7/merged overlay 75G 74G 0 100% /var/lib/docker/overlay2/2ced4e02759e7bfffcf9afc33dac88d1277d7568906a557b430406e4276f0ee4/merged overlay 75G 74G 0 100% /var/lib/docker/overlay2/95628946ec04929aae79fe8046bd20af7c801a07e7b45ce8101e065be9bebb71/merged#Help
@3246 are your backups on the same server ? Can you check
/var/backups? If you are using external backups, then it's possible that older backups are there. You can safely delete those (there will be a snapshot directory and also timestamped directories). -
@3246 are your backups on the same server ? Can you check
/var/backups? If you are using external backups, then it's possible that older backups are there. You can safely delete those (there will be a snapshot directory and also timestamped directories). -
@girish sorry, no dice. Freed up 28.1MB though
I keep backups on a StorageBox (external).
What else can I do?
-
@3246 can you try
docker image prune -a? Cloudron doesn't actually store anything outside /home/yellowtent and docker images. The mysql database is outside but it's usually very small.@girish Total reclaimed space: 6.344GB
Still showed a whopping amount of disk use, which may be OK afterall?
I resized the VM but getting lots of errors. Can someone have look please?
I can't install or remove apps
š Worried about staying safe online?
š There's a lot you can do! https://www.bebraver.online -
@girish Total reclaimed space: 6.344GB
Still showed a whopping amount of disk use, which may be OK afterall?
I resized the VM but getting lots of errors. Can someone have look please?
I can't install or remove apps
-
@3246 sure, can you send a mail to support@cloudron.io ? Not sure what is going on.
@girish "You have mail!"
š Worried about staying safe online?
š There's a lot you can do! https://www.bebraver.online -
@3246 So, the issue is that Hetzner storage box is not mounting anymore. It returns
STATUS_ACCOUNT_DISABLED. I actually hit this issue today just this morning. I have to debug further why it's not cifs mounting anymore.In previous release of Cloudron, it was not detecting whether something is mounted properly before backing up. This caused Cloudron to backup in the local filesystem instead of CIFS (when the mounting failed, which is the case now). So, just removing the local file system backup got back the 25GB or so.
In 7.3.4, you will see now that the backups page is showing the mount error. I am still debugging why it refuses to cifs mount (this also applies to hetzner storage box on our servers, it fails with same error)
-
@3246 I had to go into Hetzner storage box UI and then disable samba and enable it again. Then, in Cloudron dashboard, Backups -> Remount Storage. Does it give green icon ?
@girish thank you. I made sure it's on (looks like it was off!).
Backups won't mount though.
Access denied. Create the directory and run "chown yellowtent:yellowtent /mnt/cloudronbackup" on the serverDoing that now.
š Worried about staying safe online?
š There's a lot you can do! https://www.bebraver.online -
@girish thank you. I made sure it's on (looks like it was off!).
Backups won't mount though.
Access denied. Create the directory and run "chown yellowtent:yellowtent /mnt/cloudronbackup" on the serverDoing that now.
-
@3246 Are you able to write to us one support@ again, please? It's not clear what is happenning to me.
@girish my bad! You can blame user error for this one
Hetzner StorageBoxes need a different path when sub-accounts are used, which I forgot to note in my docu and went changing things.
Here's to anyone dealing with change management professionally, ITIL or otherwise, you are a must-have no matter how often peeps complain about 'red tape'
There's a reason why I keep documenting things but no AI or home assistant has been invented yet to make me actually remember to stick to it
Notes to self:
- Before changing critical things like backups(!) make sure to follow the documentation.
- Do not disable critical features on accounts like CIFS access (d'oh).
- Also, make sure to test changes thoroughly before committing to them.
- Also also, enjoy Cloudron and the awesome people making it a super nice place to be
