Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    Best way to upload large amounts of data to servers/Cloud

    Discuss
    6
    7
    96
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P
      privsec last edited by

      Hello,
      I have been smacking my head against a wall for the past week. I have 12 TB of data that I need uploaded to two different targets.
      Im at present working with a residential upload connection of 20mbps max upload (this will take forever, I know) but my issue is I am using cyberduck and I have connections via idrive and my server set up via SSH, however, cyberduck seems to be always having a fit after I attempt to upload more then .01 kbs (Im not kidding).

      What have you done to implement a proper backup solution?

      robi fbartels 2 Replies Last reply Reply Quote 1
      • robi
        robi @privsec last edited by

        @privsec

        1. visit a friend/place who has a blazingly fast connection. (corps, ISPs, datacenters, universities, .FTTH, ..)
        2. avoid uploading twice. (upload once, then xfer from there to the other)

        There are many clients that can access S3, so you can try those to see if they work better or use a different configuration.

        Avoid large transfers over SSH as it's inherently slow.

        Life of Advanced Technology

        1 Reply Last reply Reply Quote 1
        • fbartels
          fbartels App Dev @privsec last edited by

          @privsec https://rclone.org/ is a good tool to interact with different cloud apis, but in itself it is not a backup tool.

          1 Reply Last reply Reply Quote 2
          • G
            GregY last edited by

            FWIW, and I'm far from an expert, but I use Backblaze B2 for my backups. I previously used Wasabi but I switched to Backblaze B2 since they can mail you a physical drive with your data on it should you need to restore from backup.

            I have about 10 TB total on my NAS that I need saving and, so far, I've backed up ~9 TB of it over the last 6 or so months As fbartels suggested, rclone is a good tool for this.

            I simply SSH into my QNAP NAS, and use 'rclone sync' to upload more data. Since I also have automatic local backups - an older NAS which is located in another room from my primary NAS - I don't need to upload everything all at once to BBB2. I normally upload 30G or so per day. Once I have the bulk of it done, I'll write a bash script for my NAS that will run in the middle of the night to automate my backups.

            Using rclone takes some trial and error to make sure you're uploading what you want, when you want. Luckily there's a --dry-run flag that will simulate your backup without actually modifying files or uploading any data.

            I also use Backblaze B2 as my cloudron backup option.

            1 Reply Last reply Reply Quote 5
            • P
              privsec last edited by

              Thanks all, Ill look into these options

              1 Reply Last reply Reply Quote 0
              • marcusquinn
                marcusquinn last edited by

                • https://www.backblaze.com/b2/solutions/datatransfer/fireball.html

                Perhaps others offer similar. If you can get the storage temporarily onto something with a faster connection, that's probably your cheapest route.

                We're not here for a long time - but we are here for a good time :)
                Jersey/UK
                Work & Ecommerce Advice: https://brandlight.org
                Personal & Software Tips: https://marcusquinn.com

                jdaviescoates 1 Reply Last reply Reply Quote 1
                • jdaviescoates
                  jdaviescoates @marcusquinn last edited by

                  @marcusquinn said in Best way to upload large amounts of data to servers/Cloud:

                  Perhaps others offer similar

                  I'm pretty sure you used to be able to post Amazon DVDs or hard drives full of data, but not sure you still can and couldn't find anything about after a quick search

                  I use Cloudron with Gandi & Hetzner

                  1 Reply Last reply Reply Quote 0
                  • First post
                    Last post
                  Powered by NodeBB