Disk full after update

[3246]

My box was prompting me to reboot after the latest package updates and now refuses to resurface because it's out of disk space

According to my research, the disk space used should be 46G out of 75G but df -h shows 75G used by the overlay file system(s):

root@my:/# du -ch -t 1024m
15G     ./home/yellowtent/boxdata/mail/vmail/****/mail
15G     ./home/yellowtent/boxdata/mail/vmail/****
15G     ./home/yellowtent/boxdata/mail/vmail
15G     ./home/yellowtent/boxdata/mail
15G     ./home/yellowtent/boxdata
1.7G    ./home/yellowtent/platformdata/mysql/2e6eb3bd9d3a9626
2.0G    ./home/yellowtent/platformdata/mysql
1.2G    ./home/yellowtent/platformdata/logs
4.0G    ./home/yellowtent/platformdata
1.7G    ./home/yellowtent/appsdata
21G     ./home/yellowtent
21G     ./home
1.3G    ./var/lib/docker/volumes/eb0fd2ce95a38087ad725cf43ba1479e74bfa90f1ca29be2cecdb5d212e7a00b/_data/collaboraoffice
1.4G    ./var/lib/docker/volumes/eb0fd2ce95a38087ad725cf43ba1479e74bfa90f1ca29be2cecdb5d212e7a00b/_data
1.4G    ./var/lib/docker/volumes/eb0fd2ce95a38087ad725cf43ba1479e74bfa90f1ca29be2cecdb5d212e7a00b
1.3G    ./var/lib/docker/volumes/43e615fdc4e362f869254b9c53fd70b5db008d7cc6dcc39d450735d77851ad46/_data/collaboraoffice
1.4G    ./var/lib/docker/volumes/43e615fdc4e362f869254b9c53fd70b5db008d7cc6dcc39d450735d77851ad46/_data
1.4G    ./var/lib/docker/volumes/43e615fdc4e362f869254b9c53fd70b5db008d7cc6dcc39d450735d77851ad46
3.7G    ./var/lib/docker/volumes
1.1G    ./var/lib/docker/overlay2/06619deef5372d52ad33f368ae13989db4af9d88c284fb2bdd3a7cd22f64fdaf/merged/usr/lib
2.2G    ./var/lib/docker/overlay2/06619deef5372d52ad33f368ae13989db4af9d88c284fb2bdd3a7cd22f64fdaf/merged/usr
2.6G    ./var/lib/docker/overlay2/06619deef5372d52ad33f368ae13989db4af9d88c284fb2bdd3a7cd22f64fdaf/merged
2.6G    ./var/lib/docker/overlay2/06619deef5372d52ad33f368ae13989db4af9d88c284fb2bdd3a7cd22f64fdaf
1.1G    ./var/lib/docker/overlay2/10455ab22a1a211573f5caaf95ed2bb087447252b65489b21ff61ef45d145351/merged/usr/lib
2.2G    ./var/lib/docker/overlay2/10455ab22a1a211573f5caaf95ed2bb087447252b65489b21ff61ef45d145351/merged/usr
2.3G    ./var/lib/docker/overlay2/10455ab22a1a211573f5caaf95ed2bb087447252b65489b21ff61ef45d145351/merged
2.3G    ./var/lib/docker/overlay2/10455ab22a1a211573f5caaf95ed2bb087447252b65489b21ff61ef45d145351
1.9G    ./var/lib/docker/overlay2/7cc0ed857f244984e86fb2ba98a894a14512fe77bc702465adba6a3ca7fb8487/diff/app/code/packages
2.6G    ./var/lib/docker/overlay2/7cc0ed857f244984e86fb2ba98a894a14512fe77bc702465adba6a3ca7fb8487/diff/app/code
2.6G    ./var/lib/docker/overlay2/7cc0ed857f244984e86fb2ba98a894a14512fe77bc702465adba6a3ca7fb8487/diff/app
2.9G    ./var/lib/docker/overlay2/7cc0ed857f244984e86fb2ba98a894a14512fe77bc702465adba6a3ca7fb8487/diff
2.9G    ./var/lib/docker/overlay2/7cc0ed857f244984e86fb2ba98a894a14512fe77bc702465adba6a3ca7fb8487
1.1G    ./var/lib/docker/overlay2/6d96c852b5d2b8a33e76cccc6f9a4727fa46a982fc567e34d87cde34949616be/diff/usr/lib
1.6G    ./var/lib/docker/overlay2/6d96c852b5d2b8a33e76cccc6f9a4727fa46a982fc567e34d87cde34949616be/diff/usr
1.7G    ./var/lib/docker/overlay2/6d96c852b5d2b8a33e76cccc6f9a4727fa46a982fc567e34d87cde34949616be/diff
1.7G    ./var/lib/docker/overlay2/6d96c852b5d2b8a33e76cccc6f9a4727fa46a982fc567e34d87cde34949616be
1.4G    ./var/lib/docker/overlay2/95628946ec04929aae79fe8046bd20af7c801a07e7b45ce8101e065be9bebb71/merged/usr/local/share/.cache/yarn/v6
1.4G    ./var/lib/docker/overlay2/95628946ec04929aae79fe8046bd20af7c801a07e7b45ce8101e065be9bebb71/merged/usr/local/share/.cache/yarn
1.4G    ./var/lib/docker/overlay2/95628946ec04929aae79fe8046bd20af7c801a07e7b45ce8101e065be9bebb71/merged/usr/local/share/.cache
1.4G    ./var/lib/docker/overlay2/95628946ec04929aae79fe8046bd20af7c801a07e7b45ce8101e065be9bebb71/merged/usr/local/share
1.9G    ./var/lib/docker/overlay2/95628946ec04929aae79fe8046bd20af7c801a07e7b45ce8101e065be9bebb71/merged/usr/local
1.1G    ./var/lib/docker/overlay2/95628946ec04929aae79fe8046bd20af7c801a07e7b45ce8101e065be9bebb71/merged/usr/lib
3.6G    ./var/lib/docker/overlay2/95628946ec04929aae79fe8046bd20af7c801a07e7b45ce8101e065be9bebb71/merged/usr
4.6G    ./var/lib/docker/overlay2/95628946ec04929aae79fe8046bd20af7c801a07e7b45ce8101e065be9bebb71/merged
4.6G    ./var/lib/docker/overlay2/95628946ec04929aae79fe8046bd20af7c801a07e7b45ce8101e065be9bebb71
1.1G    ./var/lib/docker/overlay2/cd8242f93484524500811c52c3db963afce7af67e1402c1d0009c597878910d7/merged/usr/lib
2.2G    ./var/lib/docker/overlay2/cd8242f93484524500811c52c3db963afce7af67e1402c1d0009c597878910d7/merged/usr
du: fts_read failed: ./var/lib/docker/overlay2/cd8242f93484524500811c52c3db963afce7af67e1402c1d0009c597878910d7/merged: No such file or directory
46G     total

yet

root@my:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            1.9G     0  1.9G   0% /dev
tmpfs           384M  2.5M  382M   1% /run
/dev/sda1        75G   74G     0 100% /
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda15      253M  5.2M  247M   3% /boot/efi
overlay          75G   74G     0 100% /var/lib/docker/overlay2/06619deef5372d52ad33f368ae13989db4af9d88c284fb2bdd3a7cd22f64fdaf/merged
overlay          75G   74G     0 100% /var/lib/docker/overlay2/f4c82f7c6f800a006af5c1f3f0c9e5a04d2e91ea1116df30f82b03fb72d5f4ac/merged
overlay          75G   74G     0 100% /var/lib/docker/overlay2/10455ab22a1a211573f5caaf95ed2bb087447252b65489b21ff61ef45d145351/merged
overlay          75G   74G     0 100% /var/lib/docker/overlay2/f39da0d113262407c436b4f8a5952c94bce32aa550eff18bdb6791df757fb259/merged
overlay          75G   74G     0 100% /var/lib/docker/overlay2/6f7dd5920dd9595ef626d0531ccd8eb57bc8fa1d09ff3c9ba7bdab354a6a834a/merged
overlay          75G   74G     0 100% /var/lib/docker/overlay2/c77865d8d5bf4f7149f0a20a2ea4078e9ebe5aa7ae539daa8b1c5790272f5089/merged
overlay          75G   74G     0 100% /var/lib/docker/overlay2/e5e74cd6999b21caca6632a4b2e1be0b0c1c5665d9a77e25c97d223106ca9e9f/merged
overlay          75G   74G     0 100% /var/lib/docker/overlay2/7d7e953ff32324575fd42f9bed6f41f1413a68e78a109dbe3fee57aaefbae05e/merged
overlay          75G   74G     0 100% /var/lib/docker/overlay2/d31fc02ad295d6045e24fbf1349df485d6c6616fb48ffa6a9b440af3bc772ed2/merged
overlay          75G   74G     0 100% /var/lib/docker/overlay2/9ed98d68929478b7c5617fb912de54b028d0cb9c4026882c5e62af7a9dc6f1b7/merged
overlay          75G   74G     0 100% /var/lib/docker/overlay2/761d1029ed47fe60c26b91c30d620d213c135e22dad486e75c1aeb13fa12bf96/merged
overlay          75G   74G     0 100% /var/lib/docker/overlay2/1afa4fe659b317a5a004f2d6871df3b1b11af62ac22bdb495a9492e5fd42a4bf/merged
overlay          75G   74G     0 100% /var/lib/docker/overlay2/d21bcb497a014767b3fa030bb295467f90e46d4d45907482b82c4c1eb5ffa33d/merged
overlay          75G   74G     0 100% /var/lib/docker/overlay2/7fade2e1d24f73d71c01a8aceb16e1a144f6419b11272f4618ba7394622ef2bb/merged
overlay          75G   74G     0 100% /var/lib/docker/overlay2/09b7f60915e418a06e19310c28a0161444f486d86f977fa822b34d11315b2735/merged
overlay          75G   74G     0 100% /var/lib/docker/overlay2/e52d8bfa8c01be3c61b64a880a8a35222a853ed28e62f392d9e7b7e5f4cac568/merged
tmpfs           384M     0  384M   0% /run/user/0
overlay          75G   74G     0 100% /var/lib/docker/overlay2/cd8242f93484524500811c52c3db963afce7af67e1402c1d0009c597878910d7/merged
overlay          75G   74G     0 100% /var/lib/docker/overlay2/2ced4e02759e7bfffcf9afc33dac88d1277d7568906a557b430406e4276f0ee4/merged
overlay          75G   74G     0 100% /var/lib/docker/overlay2/95628946ec04929aae79fe8046bd20af7c801a07e7b45ce8101e065be9bebb71/merged

#Help

[girish]

@3246 are your backups on the same server ? Can you check /var/backups ? If you are using external backups, then it's possible that older backups are there. You can safely delete those (there will be a snapshot directory and also timestamped directories).

[3246]

@girish sorry, no dice. Freed up 28.1MB though

I keep backups on a StorageBox (external).

What else can I do?

[girish]

@3246 can you try docker image prune -a ? Cloudron doesn't actually store anything outside /home/yellowtent and docker images. The mysql database is outside but it's usually very small.

[3246]

@girish Total reclaimed space: 6.344GB

Still showed a whopping amount of disk use, which may be OK afterall?

I resized the VM but getting lots of errors. Can someone have look please?

I can't install or remove apps

[girish]

@3246 sure, can you send a mail to support@cloudron.io ? Not sure what is going on.

[3246]

@girish "You have mail!"

[girish]

@3246 So, the issue is that Hetzner storage box is not mounting anymore. It returns STATUS_ACCOUNT_DISABLED . I actually hit this issue today just this morning. I have to debug further why it's not cifs mounting anymore.

In previous release of Cloudron, it was not detecting whether something is mounted properly before backing up. This caused Cloudron to backup in the local filesystem instead of CIFS (when the mounting failed, which is the case now). So, just removing the local file system backup got back the 25GB or so.

In 7.3.4, you will see now that the backups page is showing the mount error. I am still debugging why it refuses to cifs mount (this also applies to hetzner storage box on our servers, it fails with same error)

[girish]

@3246 I had to go into Hetzner storage box UI and then disable samba and enable it again. Then, in Cloudron dashboard, Backups -> Remount Storage. Does it give green icon ?

[3246]

@girish thank you. I made sure it's on (looks like it was off!).

Backups won't mount though.

Access denied. Create the directory and run "chown yellowtent:yellowtent /mnt/cloudronbackup" on the server

Doing that now.

[3246]

@3246 recreated the folder and chown'd it. Still won't remount and no error given

[girish]

@3246 Are you able to write to us one support@ again, please? It's not clear what is happenning to me.

[3246]

@girish my bad! You can blame user error for this one

Hetzner StorageBoxes need a different path when sub-accounts are used, which I forgot to note in my docu and went changing things.

Here's to anyone dealing with change management professionally, ITIL or otherwise, you are a must-have no matter how often peeps complain about 'red tape'

There's a reason why I keep documenting things but no AI or home assistant has been invented yet to make me actually remember to stick to it

Notes to self:

• Before changing critical things like backups(!) make sure to follow the documentation.
• Do not disable critical features on accounts like CIFS access (d'oh).
• Also, make sure to test changes thoroughly before committing to them.
• Also also, enjoy Cloudron and the awesome people making it a super nice place to be