Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    Updates for apps sorted by size

    Feature Requests
    8
    38
    668
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • jdaviescoates
      jdaviescoates last edited by

      That does sound insanely expensive!

      Just to also add, I'm doing daily back-ups of 49 apps that total 247.12 GB (plus a few other small Cloudrons too) and my backups (all tarball backups) comes to a total of 3.63 TB and I've got all that stored on a 5TB Hetzner that only costs €13.08/mo

      I use Cloudron with Gandi & Hetzner

      subven 1 Reply Last reply Reply Quote 0
      • subven
        subven @jdaviescoates last edited by

        @jdaviescoates said in Updates for apps sorted by size:

        That does sound insanely expensive!

        Just to also add, I'm doing daily back-ups of 49 apps that total 247.12 GB (plus a few other small Cloudrons too) and my backups (all tarball backups) comes to a total of 3.63 TB and I've got all that stored on a 5TB Hetzner that only costs €13.08/mo

        Same for me ^^ I also use the same storagebox for my Laptop, mobile and Desktop backups mostly done with Duplicati.

        As my Cloudron grows I want to switch over to RSYNC because at the moment, TGZ is still faster because of many, many, many small files.

        1 Reply Last reply Reply Quote 0
        • marcusquinn
          marcusquinn last edited by

          Paid $20 for 5TB for a year with idrive.com/e2 sign-up offer.

          We're not here for a long time - but we are here for a good time :)
          Jersey/UK
          Work & Ecommerce Advice: https://brandlight.org
          Personal & Software Tips: https://marcusquinn.com

          jdaviescoates J 2 Replies Last reply Reply Quote 1
          • jdaviescoates
            jdaviescoates @marcusquinn last edited by

            @marcusquinn said in Updates for apps sorted by size:

            Paid $20 for 5TB for a year with idrive.com/e2 sign-up offer.

            Those are still the prices too:

            467267a4-210a-4eaa-832b-4e2daa6988c1-image.png

            That is insanely cheap for the first year, although I note their prices are slightly higher than Hetzner after that first year.

            I use Cloudron with Gandi & Hetzner

            P 1 Reply Last reply Reply Quote 1
            • P
              privsec last edited by

              This is partly why I want to know what apps have what amount of space in backups.

              I have 5.6 tb of data in wasabi, but I have no idea as to why its that high d98a0481-8060-4669-a745-af4a1db844bd-image.png

              It does look like my backups in cloudron do not list all the backups in Wasabi

              6ef4af76-f44a-45d6-af18-36de8b8df3a6-image.png

              MY thought process is that this is an issue/bug

              jdaviescoates marcusquinn subven 4 Replies Last reply Reply Quote 0
              • jdaviescoates
                jdaviescoates @privsec last edited by

                @privsec I just want to make sure you've seen this post?

                @marcusquinn said in Updates for apps sorted by size:

                @privsec Ouch, that's way too much for personal use!

                1. You must make sure that version history is OFF for ALL buckets.
                2. They do have a 3 month data retention charge, so even if you do reduce storage, you'll still be paying for the highest amount used for another 3 months.
                3. Make sure all your backups are Tarball, it uses a lot less disk space, and is just as fast, if not faster to upload.
                4. Checkout idrive.com/e2, I've moved from Wasabi to that now, and happy with it for cost, especially with Tarball backups being that much less in disk space. Again, you don't need version history on any backups, that would just be duplicate data that Cloudron already covers with your retention settings.

                If you haven't turned of version history for all your buckets this is likely the cause.

                I use Cloudron with Gandi & Hetzner

                P 1 Reply Last reply Reply Quote 1
                • marcusquinn
                  marcusquinn @privsec last edited by

                  @privsec Sounds like it is because you have Wasabi bucket versioning on, which is basically ballooning your account every time you backup, as it's making copies of ever file, every time. Been there, had the same issue, they are a bitch for having that gotchya setting by default, and they also won't let you delete your card details - ever - so only way not to pay on leaving them is cancelling your card. Even then, I have a feeling they'd get threatening.

                  We're not here for a long time - but we are here for a good time :)
                  Jersey/UK
                  Work & Ecommerce Advice: https://brandlight.org
                  Personal & Software Tips: https://marcusquinn.com

                  1 Reply Last reply Reply Quote 1
                  • marcusquinn
                    marcusquinn @privsec last edited by

                    @privsec Not a Cloudron issue or bug - its a Wasabi business model and default settings costly annoyance. Sorry you found this out the hard way. I've tried to highlight a few times around the forum.

                    We're not here for a long time - but we are here for a good time :)
                    Jersey/UK
                    Work & Ecommerce Advice: https://brandlight.org
                    Personal & Software Tips: https://marcusquinn.com

                    1 Reply Last reply Reply Quote 0
                    • P
                      privsec @jdaviescoates last edited by

                      @jdaviescoates said in Updates for apps sorted by size:

                      @privsec I just want to make sure you've seen this post?

                      @marcusquinn said in Updates for apps sorted by size:

                      @privsec Ouch, that's way too much for personal use!

                      1. You must make sure that version history is OFF for ALL buckets.
                      2. They do have a 3 month data retention charge, so even if you do reduce storage, you'll still be paying for the highest amount used for another 3 months.
                      3. Make sure all your backups are Tarball, it uses a lot less disk space, and is just as fast, if not faster to upload.
                      4. Checkout idrive.com/e2, I've moved from Wasabi to that now, and happy with it for cost, especially with Tarball backups being that much less in disk space. Again, you don't need version history on any backups, that would just be duplicate data that Cloudron already covers with your retention settings.

                      If you haven't turned of version history for all your buckets this is likely the cause.

                      @marcusquinn said in Updates for apps sorted by size:

                      @privsec Sounds like it is because you have Wasabi bucket versioning on, which is basically ballooning your account every time you backup, as it's making copies of ever file, every time. Been there, had the same issue, they are a bitch for having that gotchya setting by default, and they also won't let you delete your card details - ever - so only way not to pay on leaving them is cancelling your card. Even then, I have a feeling they'd get threatening.

                      @marcusquinn said in Updates for apps sorted by size:

                      @privsec Not a Cloudron issue or bug - its a Wasabi business model and default settings costly annoyance. Sorry you found this out the hard way. I've tried to highlight a few times around the forum.

                      I have confirmed versioning is not turned on
                      d6513102-582a-43d4-abfa-8e3c5024168a-image.png

                      This is my utilization overview
                      97922aa0-f7e7-4de3-84dc-66ddac29a813-image.png

                      1 Reply Last reply Reply Quote 0
                      • subven
                        subven @privsec last edited by subven

                        @privsec said in Updates for apps sorted by size:

                        I have 5.6 tb of data in wasabi, but I have no idea as to why its that high

                        Just purge (empty) your Cloudron bucket at Wasabi. Utilisation for your account should be 0 now. Make a (manual) backup afterwards. If you use TGZ, your app backups are compressed and it should be easy to determine individual size. Also check back overall utilisation of storage within your Wasabi. Now make another backup and compare the groth per backup done. If you keep track on this process over lets say a week, you will find out whats going wrong.

                        PS: How many data does your Cloudron + apps store? You can find out at System Info.

                        EDIT: I guess I found the answer to your question.

                        @privsec said in Updates for apps sorted by size:

                        This is my utilization overview

                        Take a look at this: How does Wasabi's minimum storage duration policy work?

                        I did not read the page but at first glance I can safely assume what is going on 😉 Maybe move away from Wasabi and use other known good services like Hetzner Storagebox. In case it is not obvious: Wasabi lets you pay for data you did already delete (the black bar in the graph) in addition to data you store at the moment (green bar).

                        P 2 Replies Last reply Reply Quote 0
                        • P
                          privsec @subven last edited by

                          @subven

                          Disk Utilization
                          977ca536-9ebc-485d-b10e-02bfd130b480-image.png

                          Encryption settings
                          9a6c06bf-e08b-447a-a9af-c4180f59773d-image.png

                          P 1 Reply Last reply Reply Quote 0
                          • P
                            privsec @subven last edited by

                            @subven
                            I understand that Cloudron is deleting backups and Wasabi is charging for that, however, thats only part of the issue.

                            My backup size though should not be anywhere nere 6 TBs even if an exact image copy of my server is uploaded to wasabi at each backup.

                            Backup settings
                            dd5837bc-d059-4e4d-8354-449dc9ecd1f1-image.png

                            The problem will simply follow me around until I discern why Cloudron has so many files "backed up" when according to cloudron, not many are.

                            subven marcusquinn 2 Replies Last reply Reply Quote 0
                            • subven
                              subven @privsec last edited by

                              @privsec said in Updates for apps sorted by size:

                              My backup size though should not be anywhere nere 6 TBs

                              Where do you get the 6TB number from? Is it the green bar from your Wasabi account? Go into your invoices at Wasabi and take a look at the Deleted Storage vs Total Storage numbers.

                              Also: 4 backups per month = 4 x ~500GB + extra backups for plattform and app backups. This could be something around 3-4TB in actual backup size + even more "dead" (deleted) storage you use because of the retention policy.

                              1 Reply Last reply Reply Quote 0
                              • subven
                                subven last edited by subven

                                One of my main Cloudrons with ~75GB in appdata including mail + box.

                                2023-02-02 05_22_11-Window.png

                                Daily backups with 1 week retention.

                                2023-02-02 05_22_32-Window.png

                                Backup storage usage is 552GB in total. Some folders are older then they should be but they do not contain full backups as I also have stopped apps.
                                sewwfwezte4wwz.png

                                Total size for the "out of scope" folders is 57GB.

                                segfererhg.png

                                Lets do the math! Assumed backup size = 75GB * 7 days = 525 GB (without compression).
                                If I factor in compression it is 55-60GB * 7 days = 385-420 GB.

                                Real size 552GB so it is around 1/3 more then I would assume. This seems fine to me since additional plattform backups and app backups before updates are also included.

                                btw. this is from a 5TB Hetzner storage box 🙂

                                1 Reply Last reply Reply Quote 0
                                • marcusquinn
                                  marcusquinn @privsec last edited by marcusquinn

                                  @privsec Seriously, get yourself off of Wasabi, OK service, but sharks for bill shock. You could get yourself setup with IDrive E2, change to Tarball backups, and delete all your Wasabi buckets in the time it's taking to go through all this diagnostics. It's not a Cloudron issue, it's a Wasabi trickery issue, their interface is shit, their retention policy is annoying, their manners were never that helpful when I had the same issue.

                                  We're not here for a long time - but we are here for a good time :)
                                  Jersey/UK
                                  Work & Ecommerce Advice: https://brandlight.org
                                  Personal & Software Tips: https://marcusquinn.com

                                  timconsidine P 2 Replies Last reply Reply Quote 3
                                  • timconsidine
                                    timconsidine App Dev @marcusquinn last edited by

                                    @marcusquinn said in Updates for apps sorted by size:

                                    Seriously, get yourself off of Wasabi, OK service, but sharks for bill shock

                                    Absolutely
                                    I stopped using Wasabi and use Scaleway

                                    1 Reply Last reply Reply Quote 3
                                    • P
                                      privsec @marcusquinn last edited by

                                      @marcusquinn I get it, totally. And I plan on it.
                                      But the original issue remains.

                                      Even If I switch services, without versioning I have nearly 6TB of data being stored.

                                      I want to know where that's from so I can prune as needed.

                                      subven jdaviescoates 2 Replies Last reply Reply Quote 0
                                      • subven
                                        subven @privsec last edited by subven

                                        @privsec said in Updates for apps sorted by size:

                                        I want to know where that's from so I can prune as needed.

                                        Again...where do you get the 6TB number from? You can search through your backup files by domain like app_domain.com* to figure out what app consumes how much within your backup.

                                        rezhriujrt6zzjiktz.png

                                        So in my case, 25GB compressed appdata ending up as 275GB in my backup. 1 backup per day and 7 days retention. Keep in mind that another another full mirror is stored in the backups "storage" folder under different naming.

                                        Because you have one big app (the light blue one...is it Nextcloud?) with lots of data in it, this could be the reason for your backup size. Also consider using RSYNC if you have lots of huge files that don't change that often.

                                        1 Reply Last reply Reply Quote 0
                                        • jdaviescoates
                                          jdaviescoates @privsec last edited by

                                          @privsec said in Updates for apps sorted by size:

                                          Even If I switch services, without versioning I have nearly 6TB of data being stored.

                                          I bet you wouldn't.

                                          And as others have explained, just do tarball backups one by one on a new backup provider and then you'll know how much space each app's backup takes up in reality.

                                          I use Cloudron with Gandi & Hetzner

                                          1 Reply Last reply Reply Quote 0
                                          • P
                                            privsec @jdaviescoates last edited by

                                            @subven

                                            @privsec said in Updates for apps sorted by size:

                                            This is partly why I want to know what apps have what amount of space in backups.

                                            I have 5.6 tb of data in wasabi, but I have no idea as to why its that high d98a0481-8060-4669-a745-af4a1db844bd-image.png

                                            It does look like my backups in cloudron do not list all the backups in Wasabi

                                            6ef4af76-f44a-45d6-af18-36de8b8df3a6-image.png

                                            MY thought process is that this is an issue/bug

                                            1 Reply Last reply Reply Quote 0
                                            • P
                                              privsec @privsec last edited by

                                              @privsec said in Updates for apps sorted by size:

                                              @subven

                                              Disk Utilization
                                              977ca536-9ebc-485d-b10e-02bfd130b480-image.png

                                              Encryption settings
                                              9a6c06bf-e08b-447a-a9af-c4180f59773d-image.png

                                              @jdaviescoates said in Updates for apps sorted by size:

                                              @privsec said in Updates for apps sorted by size:

                                              Even If I switch services, without versioning I have nearly 6TB of data being stored.

                                              I bet you wouldn't.

                                              And as others have explained, just do tarball backups one by one on a new backup provider and then you'll know how much space each app's backup takes up in reality.

                                              I am currently using tarball for backups, am I missing something?

                                              jdaviescoates 1 Reply Last reply Reply Quote 0
                                              • jdaviescoates
                                                jdaviescoates @privsec last edited by jdaviescoates

                                                @privsec said in Updates for apps sorted by size:

                                                I am currently using tarball for backups, am I missing something?

                                                Yes you are not listening to everyone saying the same thing:

                                                Wasabi does all sorts of dodgy jiggery pokery which is increasing your storage usage.

                                                Just start experimenting with another backup provider to see what your actual storage usage should be.

                                                I use Cloudron with Gandi & Hetzner

                                                P 1 Reply Last reply Reply Quote 0
                                                • P
                                                  privsec @jdaviescoates last edited by

                                                  @jdaviescoates
                                                  Ok, I guess I am just misunderstanding folks. I will post results with a different provider when possible.

                                                  jdaviescoates 1 Reply Last reply Reply Quote 1
                                                  • jdaviescoates
                                                    jdaviescoates @privsec last edited by

                                                    @privsec I get that you want to understand what was/ is actually going on, but from where I'm sitting you spending time investigating that would just be you throwing more resources down the wasabi hole 🙂

                                                    I use Cloudron with Gandi & Hetzner

                                                    P 1 Reply Last reply Reply Quote 0
                                                    • P
                                                      privsec @jdaviescoates last edited by

                                                      @jdaviescoates I have switched to idrive just now and doing a backup. My jellyfin has all the media storage within the app container (discovered this as the backup for jelly was taking forever) I thought it was in a volume.

                                                      Now I will be looking at migrating this data to a volume.

                                                      jdaviescoates marcusquinn 2 Replies Last reply Reply Quote 2
                                                      • jdaviescoates
                                                        jdaviescoates @privsec last edited by

                                                        @privsec said in Updates for apps sorted by size:

                                                        My jellyfin has all the media storage within the app container (discovered this as the backup for jelly was taking forever) I thought it was in a volume.

                                                        Sounds like you've likely found the main culprit!

                                                        I use Cloudron with Gandi & Hetzner

                                                        1 Reply Last reply Reply Quote 1
                                                        • marcusquinn
                                                          marcusquinn @privsec last edited by

                                                          @privsec Good stuff, yes, Wasabi is storing duplicates of backups for 90 days after each backup, hence you are multiplying your storage use with them, even if you have backups being purged, they are showing that as still stored for another 90 days thereafter. Bit of their bullshit really, as they cannot recover deleted files, even a minute after.

                                                          We're not here for a long time - but we are here for a good time :)
                                                          Jersey/UK
                                                          Work & Ecommerce Advice: https://brandlight.org
                                                          Personal & Software Tips: https://marcusquinn.com

                                                          robi 1 Reply Last reply Reply Quote 3
                                                          • robi
                                                            robi @marcusquinn last edited by

                                                            @marcusquinn and they charge for it?

                                                            Life of Advanced Technology

                                                            marcusquinn 1 Reply Last reply Reply Quote 0
                                                            • marcusquinn
                                                              marcusquinn @robi last edited by

                                                              @robi Yeah. I can't recall any other S3 services doing this, but maybe they got the idea from somewhere, or maybe it's just their way of making their headline pricing appear more competitive and trying to protect revenue volatility. If it were me, I really don't think this could make as much difference to their bottom-lines as actual customer satisfaction and goodwill, because for all they might gain, I expect they lose as much in bitterness and loss of recommendation-level happiness.

                                                              We're not here for a long time - but we are here for a good time :)
                                                              Jersey/UK
                                                              Work & Ecommerce Advice: https://brandlight.org
                                                              Personal & Software Tips: https://marcusquinn.com

                                                              robi 1 Reply Last reply Reply Quote 0
                                                              • robi
                                                                robi @marcusquinn last edited by

                                                                @marcusquinn their support should be able to tell you knowing how the back end works. Some systems are highly optimized for injest, but less for garbage collection, which can be deferred since it causes injest disruption. If they do this at 90day intervals, that's a pretty poor architectural set of choices.

                                                                If it's not usable, it should not be charged or passed on to the customer. They'd be smarter to hide it with larger customers where it matters less than smaller ones where it's very obvious. 🤷

                                                                Life of Advanced Technology

                                                                1 Reply Last reply Reply Quote 1
                                                                • J
                                                                  jayonrails translator @marcusquinn last edited by

                                                                  @marcusquinn said in Updates for apps sorted by size:

                                                                  Paid $20 for 5TB for a year with idrive.com/e2 sign-up offer.

                                                                  I would like to try it. What settings do I have to appl to Cloudron backup settings to get it up and running?

                                                                  7E59A254-67AE-4DFA-A6AD-08E1F1035750.png

                                                                  marcusquinn 1 Reply Last reply Reply Quote 0
                                                                  • marcusquinn
                                                                    marcusquinn @jayonrails last edited by

                                                                    @jayonrails The generic "S3 API Compatible v4" is what's used for anything not in that list.

                                                                    We're not here for a long time - but we are here for a good time :)
                                                                    Jersey/UK
                                                                    Work & Ecommerce Advice: https://brandlight.org
                                                                    Personal & Software Tips: https://marcusquinn.com

                                                                    J 2 Replies Last reply Reply Quote 2
                                                                    • J
                                                                      jayonrails translator @marcusquinn last edited by

                                                                      @marcusquinn thanks a lot, I will try it!

                                                                      1 Reply Last reply Reply Quote 1
                                                                      • J
                                                                        jayonrails translator @marcusquinn last edited by

                                                                        @marcusquinn works perfectly for me. Thanks for your tips!

                                                                        marcusquinn 1 Reply Last reply Reply Quote 2
                                                                        • marcusquinn
                                                                          marcusquinn @jayonrails last edited by

                                                                          @jayonrails Cool. I recommend sticking to Tarball. Faster and takes up less space overall.

                                                                          We're not here for a long time - but we are here for a good time :)
                                                                          Jersey/UK
                                                                          Work & Ecommerce Advice: https://brandlight.org
                                                                          Personal & Software Tips: https://marcusquinn.com

                                                                          1 Reply Last reply Reply Quote 2
                                                                          • First post
                                                                            Last post
                                                                          Powered by NodeBB