Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    Backup Strategy Advice

    Discuss
    10
    35
    1326
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • L
      lucidfox last edited by

      I have Cloudron installed on a Netcup VPS, and am having trouble setting up a reliable backup strategy that works on a consistently while being cost effective.

      Some background info - backup size for the entire platfrom is currently about 850 GB, with most of the data being contributed to by user Nextcloud data. Automatic backups are currently scheduled to run once a week, and the retention policy is set to one month (any opinions on whether this is to lax?).

      I would like to use a different service provider for backups (different vendor and geographic region) and also have have encryption enabled, so that the storage provider doesn't have to be trusted.

      Initially, the backups were rsync to Wasabi. This worked pretty flawlessly. But Wasabi charge for deleting data, so even though the storage costs are very reasonable, the overall costs quickly become unreasonable.

      Then I switched to Backblaze, they don't charge for deleting data, but using rsync causes there to be a lot of API calls, with costs quickly adding up (though not as extremely as Wasabi).

      So I switched to tgz backups. This I guess is sub-optimal for having to transfer so much data every time (even though bandwidth is not at a premium or particularly limited). This worked OK a few times after increasing memory limits (up to over 6 GB), but I'm now getting task timeouts and am unable to successfully backup. I've played around with the memory and upload part size settings but still not luck, getting it stable.

      Is there anything obvious I'm missing? How do you do backups? Any advice in getting this right would be much appreciated. Ease of admin, cost, elasticity of storage to the actual size of the backups, and security would be the main considerations. Would setting up a separate Minio backup server and using Rsync be a good solution?

      marcusquinn P 2 Replies Last reply Reply Quote 0
      • subven
        subven last edited by subven

        I would recommend encrypted rsync backups. Because of the encryption, you don't have to trust your backup provider.

        Not sure but a good option would be Netcups storage spaces together with one of their Cloud vLAN producs to have decent backup speed. Cost would be around 15-25€ for 1TB per month.

        Because you said you want another provider maybe within another region, we have to consider other options. The best Idea I had so far is a seperate VPS with enough storage space running Minio. A cheap option (for smaller setups) would be some generic webspace. I also thought about Strato High Drive Business. 1TB for 15€/month but I don't know about traffic and speed.

        PS: I'm also looking for a decent cheap backup solution/provider for my 500GB Netcup server ^^

        murgero 1 Reply Last reply Reply Quote 1
        • marcusquinn
          marcusquinn @lucidfox last edited by

          @lucidfox Maybe try a Hetzner Storage Box, sounds like that might fit your needs. Encrypted, definitely. Probably Rsync from the sounds of things.

          I agree though, it's difficult to know without trying, and I've tried many of the things you have too and still not 100% sure I won't change things again for that perfect setup 🙂

          We're not here for a long time - but we are here for a good time :)
          Jersey/UK
          Work & Ecommerce Advice: https://brandlight.org
          Personal & Software Tips: https://marcusquinn.com

          1 Reply Last reply Reply Quote 1
          • girish
            girish Staff last edited by

            As already suggested, for large data, it's best to use a hard disk / storage box. Just nfs or cigs mount it . S3 storage is not ideal for large backups, especially large file count. Ideally, choose a block storage which is in same data center.

            In the future, we do plan to integrate something like restic or borg which will give us encrypted differential backups. I don't expect this in the next 3 months though.

            BrutalBirdie 1 Reply Last reply Reply Quote 6
            • BrutalBirdie
              BrutalBirdie Staff @girish last edited by BrutalBirdie

              @girish said in Backup Strategy Advice:

              As already suggested, for large data, it's best to use a hard disk / storage box. Just nfs or cigs mount it . S3 storage is not ideal for large backups, especially large file count. Ideally, choose a block storage which is in same data center.

              In the future, we do plan to integrate something like restic or borg which will give us encrypted differential backups. I don't expect this in the next 3 months though.

              https://forum.duplicati.com/t/big-comparison-borg-vs-restic-vs-arq-5-vs-duplicacy-vs-duplicati/9952

              The above link could be useful for comparison between restic and borg.
              But never trust, do your own tests. 😄

              ps: I dont know what duplicati is... I just found this comparison

              Like my work? Consider donating a beer 🍻 Cheers!

              1 Reply Last reply Reply Quote 2
              • P
                p44 translator @lucidfox last edited by p44

                @lucidfox I had same problem too with Wasabi, 26th of February. In January I switched to Wasabi to achieve better performances, reliability and stability (see below) and also for cheaper prices.

                Last week I had the surprise: billed 30$ instead 5$ for 1TB because of their policy.

                Yes, I had read their policy before, but you cannot know before how data upload and deletion will impact depending of Cloudron scheduled backups.

                So I came back to old solution: Hetzner Storage Box and CIFS mounting.

                Last January I switched to Wasabi because CIFS mounting had some unwanted unmouting problems. But now mounting points seems to be persistent and problem seems to be fixed.

                You can find info here.

                About strategy, I suggest to purchase a Hetzner Box or a EX server (Eg. EX42) where you can install Cloudron+Minio.

                1 Reply Last reply Reply Quote 2
                • marcusquinn
                  marcusquinn last edited by marcusquinn

                  I've had another thought on this, and something I'm thinking will work for our needs:

                  Hetzner and most others have daily snapshot backups, likely that those have the same resilience as any of their storage boxes, and using that has no CPU cost to your VPS, although in Hetzner's care a 20% extra cost on the VPS, which I feel has value in that there's no reliance on your own software to do this.

                  Which makes me think than any other off-site backups only need to be Weekly, for whatever retention period your needs require.

                  It doesn't solve the Wasabi 90-day charges for deleted data issue completely, or the Backblaze ingress costs, which aren't that bad after the first Rsync is complete - but it does reduce the data storage and cycling by a factor of 7, can be scheduled for weekends or days/nights when servers are least used, and still fulfils the provider and geo-replication aims.

                  We're not here for a long time - but we are here for a good time :)
                  Jersey/UK
                  Work & Ecommerce Advice: https://brandlight.org
                  Personal & Software Tips: https://marcusquinn.com

                  1 Reply Last reply Reply Quote 2
                  • marcusquinn
                    marcusquinn last edited by

                    ^This also has additional cover in that when there's App or Cloudron Updates, the App or whole system has a backup triggered before those are installed.

                    Basically, daily off-site backups are probably over-kill for almost all of us that have VPS backup snapshots enabled too, and for anyone that doesn't, a provider storage box is probable going to be more efficient for daily backups and S3 should really be more for weeklies.

                    We're not here for a long time - but we are here for a good time :)
                    Jersey/UK
                    Work & Ecommerce Advice: https://brandlight.org
                    Personal & Software Tips: https://marcusquinn.com

                    1 Reply Last reply Reply Quote 1
                    • L
                      lucidfox last edited by

                      Thanks everyone for sharing your suggestions and insights (this is a very special community, and it's nice to be part of it). I've decided to try encrypted rsync to a Hetzner StorageBox via a CIFS mount. Others seem to be having trouble maintaining the mount with Hetzner Storage Boxes, so hopefully that doesn't happen too often. Even though there might be latency issues, I think this would be better than using a Netcup NFS mount, just so that the backups go to another provider from what the VPS is on.

                      @girish It would great if Cloudron could add more robust and nuanced backup options at some point. But I can understand that it's not priority at the moment (y'all are doing a great job with Cloudron).

                      @marcusquinn Your thinking on using the VPS snapshot does make a lot of sense. In my case most of the app data is on an Netcup NFS mount, so snapshoting the server wouldn't be very useful (I doubt the addon storage is included). So I've sort off flipped that on it's head, and will try doing daily rysnc backups (retained for a week) and then weekly automatic snapshots on the Hetzner Storage Box (you get a certain number free, depending on storage capacity). This doesn't have the effect of conserving VPS resources, but should extend the backup range without adding to storage costs.

                      marcusquinn 1 Reply Last reply Reply Quote 1
                      • marcusquinn
                        marcusquinn @lucidfox last edited by marcusquinn

                        @lucidfox Sounds like a solid strategy to me. Certainly right to have a multi-provider & encrypted setup.

                        One of the biggest risks I see with anything nowadays isn't technical but Ts & Cs and provider lock-out.

                        Their platforms, their rules, they can change at any time and you can be an accidental victim of bad actors in ways you'll never imagine before it happens.

                        From what I understand storage boxes have hardware redundancy, so I don't think that would be a point of failure - most backup recovery needs are user or software caused.

                        In my experience, no-one ever needs a backup past a week old, what is usually needed is the most recent recoverable.

                        So I think start with shorter intervals to get it all working confidently, then extend to longer intervals and don't store too old for the sake of it if you don't have some regulatory needs to.

                        This is my personal Cloudron setup now (rsync encrypted to Wasabi) + the 7 daily provider snapshots:

                        f26d2f3d-a41b-4d62-a13e-998a058a6e2f-image.png

                        Happy with this for balancing costs, cruft and security.

                        We're not here for a long time - but we are here for a good time :)
                        Jersey/UK
                        Work & Ecommerce Advice: https://brandlight.org
                        Personal & Software Tips: https://marcusquinn.com

                        L P 2 Replies Last reply Reply Quote 2
                        • L
                          lucidfox @marcusquinn last edited by

                          @marcusquinn I don't have any regulatory needs. But it might make sense to reach back into the past, to a reasonable extent, in case a user accidentally deletes files and doesn't realise for a bit.

                          marcusquinn 1 Reply Last reply Reply Quote 0
                          • marcusquinn
                            marcusquinn @lucidfox last edited by

                            @lucidfox Haha, yeah, that happens - but some Apps have file-versioning or Trash features too, some allow users to flag a delete but the DB or file system doesn't delete until a sys admin purges. So you might be taking a sledgehammer to crack a nut.

                            If there's any apps in particular you think that might save you from backup exponentials and you're not sure, ping their names in here and I might or others might have a quicker answer.

                            We're not here for a long time - but we are here for a good time :)
                            Jersey/UK
                            Work & Ecommerce Advice: https://brandlight.org
                            Personal & Software Tips: https://marcusquinn.com

                            marcusquinn 1 Reply Last reply Reply Quote 0
                            • marcusquinn
                              marcusquinn @marcusquinn last edited by

                              @marcusquinn I suppose the other thought with rsync is that the retention period isn't so bad on storage space since its all incremental. Tarballs would be though.

                              So many variables to think through eh! So a worthy post and conversations to have evolve here for others I'm sure.

                              We're not here for a long time - but we are here for a good time :)
                              Jersey/UK
                              Work & Ecommerce Advice: https://brandlight.org
                              Personal & Software Tips: https://marcusquinn.com

                              1 Reply Last reply Reply Quote 1
                              • murgero
                                murgero App Dev @subven last edited by

                                @subven said in Backup Strategy Advice:

                                Because of the encryption, you don't have to trust your backup provider.

                                Even though they are encrypted and you can technically put them anywhere doesn't mean just go with any provider (god forbid someone got a hold of your keys to decrypt). You should have reasonable beliefs that the provider you choose wont do anything with your backups (encrypted or otherwise)

                                I use DigitalOcean for my backups (Dedicated hosting with OVH/Kimsufi) DO is cheap and reliable and I know, even with my encrypted data, they wont try to do anything with it.

                                --
                                https://urgero.org
                                ~ Professional Nerd. Freelance Programmer. ~
                                Matrix: @murgero:urgero.org

                                1 Reply Last reply Reply Quote 2
                                • P
                                  p44 translator @marcusquinn last edited by

                                  @marcusquinn Marcus do you have any workaround to manage Wasabi deletion policy? I mean, now maximum Cloudron retention policy is "1 month" or "forever".

                                  So in case 1) you are under 3 month Wasabi policy. In second case 2) you should remember each month to delete backups older than 3 months.

                                  @girish Can be interesting to add a "3 month" or "4 month" option?

                                  marcusquinn 1 Reply Last reply Reply Quote 1
                                  • marcusquinn
                                    marcusquinn @p44 last edited by

                                    @p44 My current workaround since figuring the 90-day ruse out is to make a new account and let the old one go 😂

                                    I'm still looking at options but might just made the backup interval longer for now and avoid tarballs.

                                    Agreed, the interval and retention options could have a few more to cover things like this.

                                    We're not here for a long time - but we are here for a good time :)
                                    Jersey/UK
                                    Work & Ecommerce Advice: https://brandlight.org
                                    Personal & Software Tips: https://marcusquinn.com

                                    P 1 Reply Last reply Reply Quote 1
                                    • P
                                      p44 translator @marcusquinn last edited by

                                      @marcusquinn LOL! 🙂 But I think it takes time, specially if you've to configure each time sub-accounts...

                                      1 Reply Last reply Reply Quote 0
                                      • L
                                        lucidfox last edited by

                                        so just to update my results here - i tried a hetzner storagebox, but the cifs storage mount was having some stability issues. and it was so painfully slow as to be practically unusable.

                                        so i switched to another option that i've seen mentioned in the forums here, which is to backup to a second cloudron using minio. and i'm happy to report that it's working fine. using an alphavps storage vps it's actually even cheaper, and it's quite convenient to have a further backup if needed.

                                        P subven 2 Replies Last reply Reply Quote 3
                                        • P
                                          p44 translator @lucidfox last edited by

                                          @lucidfox Thank's a lot for your advices! How many Cloudron instances you've to backup? With minio you use the same token for everyone?

                                          L 1 Reply Last reply Reply Quote 0
                                          • L
                                            lucidfox @p44 last edited by

                                            @p44 I only have the one cloudron to backup at the moment, but I'd imagine you can use the same token or create other buckets and tokens on the same minio server.

                                            P 1 Reply Last reply Reply Quote 0
                                            • P
                                              p44 translator @lucidfox last edited by

                                              @lucidfox Thank's Lucidfox. about cifs storage mount "stability issues", can you tell me more? Because now seems to be very stable and persistent...

                                              L 1 Reply Last reply Reply Quote 1
                                              • L
                                                lucidfox @p44 last edited by

                                                @p44 sometimes in cloudron it would say 'this is not a cifs mount' or something similiar. even though it seemed to have been mounted properly, and showed up when checking via ssh. it could just be something to do with mounting it incorrectly though. i figured it was not worth troubleshooting, because of the slow speeds.

                                                jdaviescoates P 2 Replies Last reply Reply Quote 1
                                                • jdaviescoates
                                                  jdaviescoates @lucidfox last edited by jdaviescoates

                                                  @lucidfox I just used the mount code provided by Hetzner here https://docs.hetzner.com/robot/storage-box/access/access-samba-cifs/ and have not had any problems

                                                  I use Cloudron with Gandi & Hetzner

                                                  ruihildt 1 Reply Last reply Reply Quote 3
                                                  • P
                                                    p44 translator @lucidfox last edited by

                                                    @lucidfox Could you paste here your cifs string template?

                                                    L 1 Reply Last reply Reply Quote 0
                                                    • ruihildt
                                                      ruihildt @jdaviescoates last edited by

                                                      @jdaviescoates Are you using Ubuntu 20?

                                                      jdaviescoates 1 Reply Last reply Reply Quote 0
                                                      • jdaviescoates
                                                        jdaviescoates @ruihildt last edited by

                                                        @ruihildt said in Backup Strategy Advice:

                                                        @jdaviescoates Are you using Ubuntu 20?

                                                        No. 18.04. I figured why change what's not broken 🙂

                                                        I'll likely use Ubuntu 20 the next time I do a fresh install on Cloudron, but am keeping my existing Cloudron's that I installed on 18.04 on 18.04.

                                                        I use Cloudron with Gandi & Hetzner

                                                        1 Reply Last reply Reply Quote 0
                                                        • L
                                                          lucidfox @p44 last edited by

                                                          @p44 i just followed the one in the cloudron docs (and also the hetzner docs).

                                                          cloudron:
                                                          //<server>/<remote_folder> /backups_cifs cifs uid=yellowtent,gid=yellowtent,user=<user>,pass=<pass>,iocharset=utf8,x-systemd.automount 0 0

                                                          hetzner:
                                                          //<username>.your-storagebox.de/backup /mnt/backup-server cifs iocharset=utf8,rw,credentials=/etc/backup-credentials.txt,uid=<system account>,gid=<system group>,file_mode=0660,dir_mode=0770 0 0

                                                          P 1 Reply Last reply Reply Quote 2
                                                          • subven
                                                            subven @lucidfox last edited by subven

                                                            @lucidfox said in Backup Strategy Advice:

                                                            so i switched to another option that i've seen mentioned in the forums here, which is to backup to a second cloudron using minio. and i'm happy to report that it's working fine. using an alphavps storage vps it's actually even cheaper, and it's quite convenient to have a further backup if needed.

                                                            I also did this and it's working fine. I choose a storage VPS at alphavps with 768MB RAM, 256GB HDD and 1TB bandwidth for 25€ per year. Had to resize the root partition and change the free RAM check at the Cloudron setup to be able to install Cloudron. After that I disabled local backups and installed Minio. Backup speed with rsync is okay for small setups and bandwidth should be suffisant since we're doing incremental backups.

                                                            PS: be aware that they freeze your server at bandwidth overuse

                                                            Are there any bandwidth overusage fees?
                                                            No! We understand how frustrating can bandwidth overusage fees be and we've taken a different approach, to ensure our flat and predictable pricing - when you come close to running out of bandwidth, we will send a warning emails. Should you not respond and upgrade the server, we'll suspend it automatically, until the bandwidth counter is reset.

                                                            scooke 1 Reply Last reply Reply Quote 4
                                                            • P
                                                              p44 translator @lucidfox last edited by

                                                              @lucidfox said in Backup Strategy Advice:

                                                              @p44 i just followed the one in the cloudron docs (and also the hetzner docs).

                                                              cloudron:
                                                              //<server>/<remote_folder> /backups_cifs cifs uid=yellowtent,gid=yellowtent,user=<user>,pass=<pass>,iocharset=utf8,x-systemd.automount 0 0

                                                              hetzner:
                                                              //<username>.your-storagebox.de/backup /mnt/backup-server cifs iocharset=utf8,rw,credentials=/etc/backup-credentials.txt,uid=<system account>,gid=<system group>,file_mode=0660,dir_mode=0770 0 0

                                                              Which one do you use? These above are standards strings. First one on the top should work.

                                                              Where is located source datacenter?

                                                              L 1 Reply Last reply Reply Quote 0
                                                              • L
                                                                lucidfox @p44 last edited by lucidfox

                                                                @p44 i don't use it anymore. i've switched to a storage vps on alphavps, with a minio backup.

                                                                P 1 Reply Last reply Reply Quote 1
                                                                • P
                                                                  p44 translator @lucidfox last edited by

                                                                  @lucidfox Ok ! 🙂 Hope it is working fine

                                                                  L 1 Reply Last reply Reply Quote 1
                                                                  • L
                                                                    lucidfox @p44 last edited by

                                                                    @p44 thanks, it works great.

                                                                    1 Reply Last reply Reply Quote 1
                                                                    • scooke
                                                                      scooke @subven last edited by

                                                                      @subven I'm not seeing the Minio app in the regular store... could you refresh my memory as to how to get it please?

                                                                      A life lived in fear is a life half-lived

                                                                      subven 1 Reply Last reply Reply Quote 0
                                                                      • subven
                                                                        subven @scooke last edited by

                                                                        @scooke actually it is in the normal store. No beta or special tricks required. You need at least Cloudron 5.3.0 to install it.

                                                                        2021-03-31 10_32_11-Window.png
                                                                        2021-03-31 10_32_19-Window.png

                                                                        scooke 1 Reply Last reply Reply Quote 2
                                                                        • scooke
                                                                          scooke @subven last edited by

                                                                          @subven Huh, so it is. That was weird for me, I scrolled up and down and even typed in the name, before asking you here. But now I can see it easily and clearly. Strange. Thank you!

                                                                          A life lived in fear is a life half-lived

                                                                          1 Reply Last reply Reply Quote 0
                                                                          • First post
                                                                            Last post
                                                                          Powered by NodeBB