Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    Solved Disk space usage seems incorrect on external disk

    Support
    graphs
    5
    27
    1326
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • d19dotca
      d19dotca last edited by girish

      Hello,

      It seems there may be an issue in 7.3 which I just noticed (unsure if this was an issue earlier), as the "Used" text doesn't seem to match in the graphic. Notice in the screenshot below that it says 200.3 GB is used, yet underneath it says Used is 189.49 GB (a difference of about 11 GB).

      244b4005-13a0-4e21-877a-646735272bef-image.png

      How is the usage calculated exactly?

      I assume the lower Used number is for how much Cloudron has saved, where-as the upper Used number is perhaps an overall disk spacing query. However I think from a UX point of view, this difference in calculations can cause confusion and either needs to be explained in the UI or should just match in both numbers.

      FYI - while Cloudron detects my disk size as 215.49 GB, here are the two numbers I see in terminal for it, with both base 10 and base 12 calculations:

      df -h | grep -i cloudron
      /dev/vdb1       216G  190G   16G  93% /mnt/cloudron-backups
      
      df -H | grep -i cloudron
      /dev/vdb1       232G  204G   17G  93% /mnt/cloudron-backups
      

      Perhaps these numbers can be displayed in a way that matches the output from the commands above as most people likely use either of those to calculator disk usage stats.

      Based on the output above, I suspect Cloudron is using the first command to calculate the size of the disk as that's closest, and the lower usage part seems right (closest again to the command output), which makes me wonder where the heck the 200.3 GB number is coming from.

      --
      Dustin Dauncey
      www.d19.ca

      1 Reply Last reply Reply Quote 0
      • d19dotca
        d19dotca last edited by d19dotca

        Noticed the usage graph doesn't seem to change. There's absolutely an issue here of some sorts. I cleared out the disk the other day and re-started the backups yesterday to it after replacing the disk with a larger sized one. It's only using about 48 GB according to the top calculation in the image but the underneath number is much larger and the graph seems to follow that larger number which is way off.

        @staff is it possible to get a response to this please? Hadn't seen one earlier, think this may have been missed.

        d206340d-4211-478d-baa7-ba15470484a9-image.png

        The current df -H output shows the following (used 61 GB, free 180 GB which mostly adds up as I have a 240 GB disk):

        df -H | grep cloudron
        /dev/vdb1       253G   61G  180G  26% /mnt/cloudronbackup
        

        --
        Dustin Dauncey
        www.d19.ca

        1 Reply Last reply Reply Quote 0
        • nebulon
          nebulon Staff last edited by

          The next patch release for 7.3 will include a whole new disk usage collection, which is not based on collectd anymore and allows to refresh with live data.

          d19dotca 1 Reply Last reply Reply Quote 5
          • d19dotca
            d19dotca @nebulon last edited by

            @nebulon Super happy to hear that! 🙂 Any ETA on when that will be released?

            --
            Dustin Dauncey
            www.d19.ca

            girish 1 Reply Last reply Reply Quote 0
            • girish
              girish Staff @d19dotca last edited by

              @d19dotca It looks something like this now:

              14a52884-c739-4379-8d17-3b778e91b5eb-image.png

              Among other fixes:

              • It tells you when the data was collected
              • Refresh to calcuate the current disk usage immediately
              • We use SI units (1000) for reporting disk usage. We use IEC units (1024) for memory usage. This seems to be some industry pseudo standard.
              d19dotca 2 Replies Last reply Reply Quote 2
              • d19dotca
                d19dotca @girish last edited by

                @girish said in Disk space usage seems incorrect on external disk:

                We use SI units (1000) for reporting disk usage. We use IEC units (1024) for memory usage. This seems to be some industry pseudo standard.

                Perfect, thanks Girish! The si units seem to be more accurate from what I can see. I just checked on my MacBook Pro and it seems the GUI shows 494 GB of disk space and that's what the disk size is according to df -H as well, so that matches / is easier to compare. 🙂

                --
                Dustin Dauncey
                www.d19.ca

                1 Reply Last reply Reply Quote 2
                • d19dotca
                  d19dotca @girish last edited by d19dotca

                  @girish Out of curiosity... how often will it check for updated disk sizes by default without manual intervention?

                  Btw, I think it'd be nice if the GUI showed free space too in an easy manner instead of having us do the math ourselves 😉 haha. For your consideration...

                  11a65538-b422-4d83-9833-634d243f00db-image.png

                  Or maybe it could be closer to how Apple displays it where the used bit is how you have it but then in the graph itself the free bit is the available space left?

                  06bf60bf-c17e-4a8f-bac7-2f365f125e5d-image.png

                  --
                  Dustin Dauncey
                  www.d19.ca

                  girish jdaviescoates 2 Replies Last reply Reply Quote 2
                  • girish
                    girish Staff @d19dotca last edited by girish

                    @d19dotca said in Disk space usage seems incorrect on external disk:

                    @girish Out of curiosity... how often will it check for updated disk sizes by default without manual intervention?

                    Currently, it's manual. You have to click the refresh button on the top right to trigger a computation. I guess we can make this periodic for next release, but the issue was this value is always getting "out of date" and thus we weren't sure if there is value in periodic computation (besides, this causes much disk spin).

                    d19dotca 1 Reply Last reply Reply Quote 0
                    • d19dotca
                      d19dotca @girish last edited by d19dotca

                      @girish Maybe it could be generated on page load automatically? Just a thought. But good either way. 🙂

                      Any possibility to get the free space calculated in the graph like it does on macOS for example? I think that'd be awesome.

                      --
                      Dustin Dauncey
                      www.d19.ca

                      1 Reply Last reply Reply Quote 1
                      • d19dotca
                        d19dotca last edited by

                        @girish - I just updated and there still seems like something wrong with the calculation...169 GB used at the top of the external disk but "this disk contains" only shows 156 GB used? How is there such a big difference there when all that's on the disk is backup files from Cloudron?

                        Does the external disk get recalculated too when I refresh it from the top?

                        5c301ea8-a08d-4a1a-98ab-f9ad08529f0b-image.png

                        --
                        Dustin Dauncey
                        www.d19.ca

                        1 Reply Last reply Reply Quote 1
                        • d19dotca
                          d19dotca last edited by

                          Update:

                          For my main disk...

                          df -H shows this for vda1 / (root):

                          /dev/vda1       182G  100G   73G  58% /
                          

                          Yet Cloudron shows the following: (108 GB used)

                          e238673c-f8f3-4a1c-8c27-49df7d765c28-image.png

                          Where is it getting that extra nearly 9 GB used from? The total disk size is close enough (181.37 GB vs 182 GB shown from command), it's just the usage part that seems quite inaccurate, especially if we're using si units this time with version 7.3.2.


                          For my backup disk it's similar situation:

                          df -H shows this for vdb1 (cloudronbackup):

                          /dev/vdb1       253G  205G   36G  86% /mnt/cloudronbackup
                          

                          Yet Cloudron shows this: (217 GB used above graph, 204 GB used below)
                          7b0b8d73-f2ac-4113-8281-bc095afffc94-image.png

                          In this case for the external backup disk, the lower Used number of 204.51 GB is pretty close to the 205 GB from the command, so that seems accurate, however in that case where is that extra ~12 GB coming from for the above-graph used number?

                          There seems to be a lot of discrepancies in the calculations here still, unfortunately. Cc @girish

                          --
                          Dustin Dauncey
                          www.d19.ca

                          1 Reply Last reply Reply Quote 1
                          • jdaviescoates
                            jdaviescoates @d19dotca last edited by

                            @d19dotca said in Disk space usage seems incorrect on external disk:

                            Btw, I think it'd be nice if the GUI showed free space too in an easy manner instead of having us do the math ourselves

                            +1

                            Been saying this for ages

                            I use Cloudron with Gandi & Hetzner

                            1 Reply Last reply Reply Quote 1
                            • Topic has been marked as a question  girish girish 
                            • Topic has been marked as solved  girish girish 
                            • d19dotca
                              d19dotca last edited by d19dotca

                              @girish - I noticed you marked this as Solved 4 days ago, but I don't believe this has actually been solved.

                              There's a pretty large discrepancy in usage for my external disk as seen in earlier screenshots, and I don't see that having been resolved yet.

                              For example, "used" under the "this disk contains:" section is 204 GB, but then the graph part shows 217 GB used. Seems like a defect that there's multiple different "used" numbers presented, and by quite a few GBs too (approximately 5 - 20 GB discrepancy).

                              Even in your own screenshot from earlier, the total underneath the graph adds up to about 36 GB but yet above the graph it shows 41 GB used (this is the minimum difference I've seen so far but still a large 5 GB difference in used space).

                              --
                              Dustin Dauncey
                              www.d19.ca

                              jdaviescoates 1 Reply Last reply Reply Quote 1
                              • jdaviescoates
                                jdaviescoates @d19dotca last edited by

                                @d19dotca said in Disk space usage seems incorrect on external disk:

                                @girish - I noticed you marked this as Solved 4 days ago, but I don't believe this has actually been solved.

                                Also:

                                it'd be nice if the GUI showed free space too in an easy manner instead of having us do the math ourselves

                                😉

                                I use Cloudron with Gandi & Hetzner

                                1 Reply Last reply Reply Quote 3
                                • Topic has been marked as unsolved  girish girish 
                                • girish
                                  girish Staff last edited by

                                  @d19dotca Can you give us the output of df -B1 --output=source,fstype,size,used,avail,pcent,target ? This is the command used to get disk information.

                                  The du information is got from the du command. Can you check if running du -DsB1 <path> matches with what you see in graph for the apps and backup directory ?

                                  Finally, for docker itself, the size comes from docker system df output.

                                  jdaviescoates d19dotca 2 Replies Last reply Reply Quote 1
                                  • jdaviescoates
                                    jdaviescoates @girish last edited by

                                    @girish are you going to add the space available?!?! 🙂 🤞

                                    I use Cloudron with Gandi & Hetzner

                                    girish 1 Reply Last reply Reply Quote 0
                                    • girish
                                      girish Staff @jdaviescoates last edited by

                                      @jdaviescoates yes, your message has been relayed to @nebulon 😄

                                      jdaviescoates 1 Reply Last reply Reply Quote 1
                                      • jdaviescoates
                                        jdaviescoates @girish last edited by

                                        @girish @nebulon thanks!

                                        I use Cloudron with Gandi & Hetzner

                                        1 Reply Last reply Reply Quote 0
                                        • nebulon
                                          nebulon Staff last edited by

                                          5e7c142d-c985-4d34-8a43-0b706b3899b7-image.png

                                          The remaining available disk space is now displayed in the remainder section of the bar.

                                          robi 1 Reply Last reply Reply Quote 3
                                          • robi
                                            robi @nebulon last edited by

                                            @nebulon what happens if it fills up and shrinks the available space to print the numbers?

                                            Life of Advanced Technology

                                            nebulon 1 Reply Last reply Reply Quote 1
                                            • d19dotca
                                              d19dotca @girish last edited by d19dotca

                                              @girish said in Disk space usage seems incorrect on external disk:

                                              @d19dotca Can you give us the output of df -B1 --output=source,fstype,size,used,avail,pcent,target ? This is the command used to get disk information.

                                              The du information is got from the du command. Can you check if running du -DsB1 <path> matches with what you see in graph for the apps and backup directory ?

                                              Finally, for docker itself, the size comes from docker system df output.

                                              Hi Girish. Happy to help here. I ran the commands as requested.

                                              The output for that df command is below (my backup disk is currently full, need to expand that in a minute, haha):

                                              ubuntu@my:~$ df -B1 --output=source,fstype,size,used,avail,pcent,target
                                              Filesystem     Type        1B-blocks         Used       Avail Use% Mounted on
                                              udev           devtmpfs   4125618176            0  4125618176   0% /dev
                                              tmpfs          tmpfs       834379776      6078464   828301312   1% /run
                                              /dev/vda1      ext4     181372190720  99085045760 72606834688  58% /
                                              tmpfs          tmpfs      4171890688            0  4171890688   0% /dev/shm
                                              tmpfs          tmpfs         5242880            0     5242880   0% /run/lock
                                              tmpfs          tmpfs      4171890688            0  4171890688   0% /sys/fs/cgroup
                                              /dev/vdb1      ext4     252515119104 252498403328           0 100% /mnt/cloudronbackup
                                              /dev/loop1     squashfs     58327040     58327040           0 100% /snap/core18/2560
                                              /dev/loop2     squashfs     66322432     66322432           0 100% /snap/core20/1623
                                              /dev/loop4     squashfs     50331648     50331648           0 100% /snap/snapd/17336
                                              /dev/loop0     squashfs     58327040     58327040           0 100% /snap/core18/2566
                                              /dev/loop3     squashfs     50331648     50331648           0 100% /snap/snapd/17029
                                              /dev/loop6     squashfs     71303168     71303168           0 100% /snap/lxd/22526
                                              /dev/loop5     squashfs     71172096     71172096           0 100% /snap/lxd/22753
                                              /dev/loop7     squashfs     65011712     65011712           0 100% /snap/core20/1611
                                              tmpfs          tmpfs       834375680            0   834375680   0% /run/user/1000
                                              

                                              This seems to show an issue... if I'm understanding the output correctly, this means that I have a /dev/vda1 (root) disk size of about 181 GB with about 99 GB used, yet it's showing 108.77 GB used in the UI (screenshot included):
                                              49f9890c-4fb1-4bab-9a25-43d4d84020b0-image.png

                                              Hopefully that clarifies the issue a bit by showing the discrepancies between what's shown in the UI vs what's shown in the command line terminal on the VM.

                                              The du command output is below:

                                              ubuntu@my:~$ sudo du -DsB1 /home/yellowtent/platformdata/
                                              9498304512	/home/yellowtent/platformdata/
                                              ubuntu@my:~$ sudo du -DsB1 /home/yellowtent/boxdata/
                                              46611869696	/home/yellowtent/boxdata/
                                              

                                              Based on that output and my UI, I believe the values of each item beneath the graphs are correct, but the total numbers calculated at the top (above the graph) are incorrect, or at least the above-graph used space number seems to be incorrect.

                                              The other docker command output is below:

                                              ubuntu@my:~$ sudo docker system df
                                              TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
                                              Images          29        21        15.58GB   6.058GB (38%)
                                              Containers      94        70        0B        0B
                                              Local Volumes   606       138       1.559GB   1.02GB (65%)
                                              Build Cache     0         0         0B        0B
                                              

                                              --
                                              Dustin Dauncey
                                              www.d19.ca

                                              nebulon 1 Reply Last reply Reply Quote 0
                                              • nebulon
                                                nebulon Staff @robi last edited by

                                                @robi currently it will just hide it, if it overflows. I think in such a case one should anyways take a closer look.

                                                1 Reply Last reply Reply Quote 2
                                                • Referenced by  nebulon nebulon 
                                                • nebulon
                                                  nebulon Staff @d19dotca last edited by

                                                  @d19dotca actually I think we need a more direct look at your system to figure out what is happening there. If you like, can you enable remote SSh support for us and send us a mail with your dashboard domain to support@cloudron.io ?

                                                  d19dotca 1 Reply Last reply Reply Quote 1
                                                  • d19dotca
                                                    d19dotca @nebulon last edited by

                                                    @nebulon No problem at all. I just enabled access for you and submitted the ticket for it through the Support page in Cloudron. Hope that helps. 🙂

                                                    --
                                                    Dustin Dauncey
                                                    www.d19.ca

                                                    nebulon 1 Reply Last reply Reply Quote 1
                                                    • nebulon
                                                      nebulon Staff @d19dotca last edited by

                                                      @d19dotca so the root cause for the discrepancy is the reserved disk space for the root user on ext4. Some info on that at https://wiki.archlinux.org/title/ext4#Reserved_blocks

                                                      This is also why df shows a discrepancy between the values. I have fixed up the UI now to reflect exactly what df reports to avoid confusion.

                                                      d19dotca 1 Reply Last reply Reply Quote 4
                                                      • Topic has been marked as solved  nebulon nebulon 
                                                      • d19dotca
                                                        d19dotca @nebulon last edited by d19dotca

                                                        @nebulon just to clarify, by “ext4” are we referring to my use of that for the backup type where I did the ext4 option with the “by-uuid” loading? Or is that related to something else? Just wanting to better understand it. Regardless though, thanks for fixing it! 🙂

                                                        --
                                                        Dustin Dauncey
                                                        www.d19.ca

                                                        nebulon 1 Reply Last reply Reply Quote 0
                                                        • nebulon
                                                          nebulon Staff @d19dotca last edited by

                                                          @d19dotca ext4 is the filesystem type in this case from your root / partition and ext4 has a feature of reserved blocks for the root user. Other disks may or may not be ext4 like for volumes or backup.

                                                          1 Reply Last reply Reply Quote 1
                                                          • First post
                                                            Last post
                                                          Powered by NodeBB