Best way to upload large amounts of data to servers/Cloud
-
Hello,
I have been smacking my head against a wall for the past week. I have 12 TB of data that I need uploaded to two different targets.
Im at present working with a residential upload connection of 20mbps max upload (this will take forever, I know) but my issue is I am using cyberduck and I have connections via idrive and my server set up via SSH, however, cyberduck seems to be always having a fit after I attempt to upload more then .01 kbs (Im not kidding).What have you done to implement a proper backup solution?
-
Hello,
I have been smacking my head against a wall for the past week. I have 12 TB of data that I need uploaded to two different targets.
Im at present working with a residential upload connection of 20mbps max upload (this will take forever, I know) but my issue is I am using cyberduck and I have connections via idrive and my server set up via SSH, however, cyberduck seems to be always having a fit after I attempt to upload more then .01 kbs (Im not kidding).What have you done to implement a proper backup solution?
- visit a friend/place who has a blazingly fast connection. (corps, ISPs, datacenters, universities, .FTTH, ..)
- avoid uploading twice. (upload once, then xfer from there to the other)
There are many clients that can access S3, so you can try those to see if they work better or use a different configuration.
Avoid large transfers over SSH as it's inherently slow.
-
Hello,
I have been smacking my head against a wall for the past week. I have 12 TB of data that I need uploaded to two different targets.
Im at present working with a residential upload connection of 20mbps max upload (this will take forever, I know) but my issue is I am using cyberduck and I have connections via idrive and my server set up via SSH, however, cyberduck seems to be always having a fit after I attempt to upload more then .01 kbs (Im not kidding).What have you done to implement a proper backup solution?
-
FWIW, and I'm far from an expert, but I use Backblaze B2 for my backups. I previously used Wasabi but I switched to Backblaze B2 since they can mail you a physical drive with your data on it should you need to restore from backup.
I have about 10 TB total on my NAS that I need saving and, so far, I've backed up ~9 TB of it over the last 6 or so months As fbartels suggested, rclone is a good tool for this.
I simply SSH into my QNAP NAS, and use 'rclone sync' to upload more data. Since I also have automatic local backups - an older NAS which is located in another room from my primary NAS - I don't need to upload everything all at once to BBB2. I normally upload 30G or so per day. Once I have the bulk of it done, I'll write a bash script for my NAS that will run in the middle of the night to automate my backups.
Using rclone takes some trial and error to make sure you're uploading what you want, when you want. Luckily there's a --dry-run flag that will simulate your backup without actually modifying files or uploading any data.
I also use Backblaze B2 as my cloudron backup option.
-
Perhaps others offer similar. If you can get the storage temporarily onto something with a faster connection, that's probably your cheapest route.
Web Design https://www.evergreen.je
Development https://brandlight.org
Life https://marcusquinn.com -
Perhaps others offer similar. If you can get the storage temporarily onto something with a faster connection, that's probably your cheapest route.
@marcusquinn said in Best way to upload large amounts of data to servers/Cloud:
Perhaps others offer similar
I'm pretty sure you used to be able to post Amazon DVDs or hard drives full of data, but not sure you still can and couldn't find anything about after a quick search
