Solved Disk space usage seems incorrect on external disk
-
Hello,
It seems there may be an issue in 7.3 which I just noticed (unsure if this was an issue earlier), as the "Used" text doesn't seem to match in the graphic. Notice in the screenshot below that it says 200.3 GB is used, yet underneath it says Used is 189.49 GB (a difference of about 11 GB).
How is the usage calculated exactly?
I assume the lower Used number is for how much Cloudron has saved, where-as the upper Used number is perhaps an overall disk spacing query. However I think from a UX point of view, this difference in calculations can cause confusion and either needs to be explained in the UI or should just match in both numbers.
FYI - while Cloudron detects my disk size as 215.49 GB, here are the two numbers I see in terminal for it, with both base 10 and base 12 calculations:
df -h | grep -i cloudron /dev/vdb1 216G 190G 16G 93% /mnt/cloudron-backupsdf -H | grep -i cloudron /dev/vdb1 232G 204G 17G 93% /mnt/cloudron-backupsPerhaps these numbers can be displayed in a way that matches the output from the commands above as most people likely use either of those to calculator disk usage stats.
Based on the output above, I suspect Cloudron is using the first command to calculate the size of the disk as that's closest, and the lower usage part seems right (closest again to the command output), which makes me wonder where the heck the 200.3 GB number is coming from.
-
Noticed the usage graph doesn't seem to change. There's absolutely an issue here of some sorts. I cleared out the disk the other day and re-started the backups yesterday to it after replacing the disk with a larger sized one. It's only using about 48 GB according to the top calculation in the image but the underneath number is much larger and the graph seems to follow that larger number which is way off.
@staff is it possible to get a response to this please? Hadn't seen one earlier, think this may have been missed.
The current df -H output shows the following (used 61 GB, free 180 GB which mostly adds up as I have a 240 GB disk):
df -H | grep cloudron /dev/vdb1 253G 61G 180G 26% /mnt/cloudronbackup -
The next patch release for 7.3 will include a whole new disk usage collection, which is not based on collectd anymore and allows to refresh with live data.
-
@nebulon Super happy to hear that!
Any ETA on when that will be released?--
Dustin Dauncey
www.d19.ca -
@d19dotca It looks something like this now:
Among other fixes:
- It tells you when the data was collected
- Refresh to calcuate the current disk usage immediately
- We use SI units (1000) for reporting disk usage. We use IEC units (1024) for memory usage. This seems to be some industry pseudo standard.
-
@girish said in Disk space usage seems incorrect on external disk:
We use SI units (1000) for reporting disk usage. We use IEC units (1024) for memory usage. This seems to be some industry pseudo standard.
Perfect, thanks Girish! The si units seem to be more accurate from what I can see. I just checked on my MacBook Pro and it seems the GUI shows 494 GB of disk space and that's what the disk size is according to
df -Has well, so that matches / is easier to compare. -
@girish Out of curiosity... how often will it check for updated disk sizes by default without manual intervention?
Btw, I think it'd be nice if the GUI showed free space too in an easy manner instead of having us do the math ourselves
haha. For your consideration...
Or maybe it could be closer to how Apple displays it where the used bit is how you have it but then in the graph itself the free bit is the available space left?
--
Dustin Dauncey
www.d19.ca -
@d19dotca said in Disk space usage seems incorrect on external disk:
@girish Out of curiosity... how often will it check for updated disk sizes by default without manual intervention?
Currently, it's manual. You have to click the refresh button on the top right to trigger a computation. I guess we can make this periodic for next release, but the issue was this value is always getting "out of date" and thus we weren't sure if there is value in periodic computation (besides, this causes much disk spin).
-
@girish Maybe it could be generated on page load automatically? Just a thought. But good either way.
Any possibility to get the free space calculated in the graph like it does on macOS for example? I think that'd be awesome.
-
@girish - I just updated and there still seems like something wrong with the calculation...169 GB used at the top of the external disk but "this disk contains" only shows 156 GB used? How is there such a big difference there when all that's on the disk is backup files from Cloudron?
Does the external disk get recalculated too when I refresh it from the top?
-
Update:
For my main disk...
df -Hshows this for vda1 / (root):/dev/vda1 182G 100G 73G 58% /Yet Cloudron shows the following: (108 GB used)
Where is it getting that extra nearly 9 GB used from? The total disk size is close enough (181.37 GB vs 182 GB shown from command), it's just the usage part that seems quite inaccurate, especially if we're using si units this time with version 7.3.2.
For my backup disk it's similar situation:
df -Hshows this for vdb1 (cloudronbackup):/dev/vdb1 253G 205G 36G 86% /mnt/cloudronbackupYet Cloudron shows this: (217 GB used above graph, 204 GB used below)
In this case for the external backup disk, the lower Used number of 204.51 GB is pretty close to the 205 GB from the command, so that seems accurate, however in that case where is that extra ~12 GB coming from for the above-graph used number?
There seems to be a lot of discrepancies in the calculations here still, unfortunately. Cc @girish
-
@d19dotca said in Disk space usage seems incorrect on external disk:
Btw, I think it'd be nice if the GUI showed free space too in an easy manner instead of having us do the math ourselves
+1
Been saying this for ages
-
Topic has been marked as a question
girish -
Topic has been marked as solved girish
-
@girish - I noticed you marked this as Solved 4 days ago, but I don't believe this has actually been solved.
There's a pretty large discrepancy in usage for my external disk as seen in earlier screenshots, and I don't see that having been resolved yet.
For example, "used" under the "this disk contains:" section is 204 GB, but then the graph part shows 217 GB used. Seems like a defect that there's multiple different "used" numbers presented, and by quite a few GBs too (approximately 5 - 20 GB discrepancy).
Even in your own screenshot from earlier, the total underneath the graph adds up to about 36 GB but yet above the graph it shows 41 GB used (this is the minimum difference I've seen so far but still a large 5 GB difference in used space).
--
Dustin Dauncey
www.d19.ca -
@d19dotca said in Disk space usage seems incorrect on external disk:
@girish - I noticed you marked this as Solved 4 days ago, but I don't believe this has actually been solved.
Also:
it'd be nice if the GUI showed free space too in an easy manner instead of having us do the math ourselves
-
Topic has been marked as unsolved girish
-
@d19dotca Can you give us the output of
df -B1 --output=source,fstype,size,used,avail,pcent,target? This is the command used to get disk information.The du information is got from the du command. Can you check if running
du -DsB1 <path>matches with what you see in graph for the apps and backup directory ?Finally, for docker itself, the size comes from
docker system dfoutput. -
@girish are you going to add the space available?!?!
-
@jdaviescoates yes, your message has been relayed to @nebulon
-
-
The remaining available disk space is now displayed in the remainder section of the bar.
-
@nebulon what happens if it fills up and shrinks the available space to print the numbers?
-
@girish said in Disk space usage seems incorrect on external disk:
@d19dotca Can you give us the output of
df -B1 --output=source,fstype,size,used,avail,pcent,target? This is the command used to get disk information.The du information is got from the du command. Can you check if running
du -DsB1 <path>matches with what you see in graph for the apps and backup directory ?Finally, for docker itself, the size comes from
docker system dfoutput.Hi Girish. Happy to help here. I ran the commands as requested.
The output for that df command is below (my backup disk is currently full, need to expand that in a minute, haha):
ubuntu@my:~$ df -B1 --output=source,fstype,size,used,avail,pcent,target Filesystem Type 1B-blocks Used Avail Use% Mounted on udev devtmpfs 4125618176 0 4125618176 0% /dev tmpfs tmpfs 834379776 6078464 828301312 1% /run /dev/vda1 ext4 181372190720 99085045760 72606834688 58% / tmpfs tmpfs 4171890688 0 4171890688 0% /dev/shm tmpfs tmpfs 5242880 0 5242880 0% /run/lock tmpfs tmpfs 4171890688 0 4171890688 0% /sys/fs/cgroup /dev/vdb1 ext4 252515119104 252498403328 0 100% /mnt/cloudronbackup /dev/loop1 squashfs 58327040 58327040 0 100% /snap/core18/2560 /dev/loop2 squashfs 66322432 66322432 0 100% /snap/core20/1623 /dev/loop4 squashfs 50331648 50331648 0 100% /snap/snapd/17336 /dev/loop0 squashfs 58327040 58327040 0 100% /snap/core18/2566 /dev/loop3 squashfs 50331648 50331648 0 100% /snap/snapd/17029 /dev/loop6 squashfs 71303168 71303168 0 100% /snap/lxd/22526 /dev/loop5 squashfs 71172096 71172096 0 100% /snap/lxd/22753 /dev/loop7 squashfs 65011712 65011712 0 100% /snap/core20/1611 tmpfs tmpfs 834375680 0 834375680 0% /run/user/1000This seems to show an issue... if I'm understanding the output correctly, this means that I have a /dev/vda1 (root) disk size of about 181 GB with about 99 GB used, yet it's showing 108.77 GB used in the UI (screenshot included):
Hopefully that clarifies the issue a bit by showing the discrepancies between what's shown in the UI vs what's shown in the command line terminal on the VM.
The du command output is below:
ubuntu@my:~$ sudo du -DsB1 /home/yellowtent/platformdata/ 9498304512 /home/yellowtent/platformdata/ ubuntu@my:~$ sudo du -DsB1 /home/yellowtent/boxdata/ 46611869696 /home/yellowtent/boxdata/Based on that output and my UI, I believe the values of each item beneath the graphs are correct, but the total numbers calculated at the top (above the graph) are incorrect, or at least the above-graph used space number seems to be incorrect.
The other docker command output is below:
ubuntu@my:~$ sudo docker system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 29 21 15.58GB 6.058GB (38%) Containers 94 70 0B 0B Local Volumes 606 138 1.559GB 1.02GB (65%) Build Cache 0 0 0B 0B--
Dustin Dauncey
www.d19.ca -
@robi currently it will just hide it, if it overflows. I think in such a case one should anyways take a closer look.
-
Referenced by
nebulon
-
@d19dotca actually I think we need a more direct look at your system to figure out what is happening there. If you like, can you enable remote SSh support for us and send us a mail with your dashboard domain to support@cloudron.io ?
-
@nebulon No problem at all. I just enabled access for you and submitted the ticket for it through the Support page in Cloudron. Hope that helps.
--
Dustin Dauncey
www.d19.ca -
@d19dotca so the root cause for the discrepancy is the reserved disk space for the root user on ext4. Some info on that at https://wiki.archlinux.org/title/ext4#Reserved_blocks
This is also why
dfshows a discrepancy between the values. I have fixed up the UI now to reflect exactly whatdfreports to avoid confusion. -
Topic has been marked as solved nebulon
-
@nebulon just to clarify, by “ext4” are we referring to my use of that for the backup type where I did the ext4 option with the “by-uuid” loading? Or is that related to something else? Just wanting to better understand it. Regardless though, thanks for fixing it!
--
Dustin Dauncey
www.d19.ca -
@d19dotca ext4 is the filesystem type in this case from your root / partition and ext4 has a feature of reserved blocks for the root user. Other disks may or may not be ext4 like for volumes or backup.
