Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse

Cloudron Forum

Apps | Demo | Docs | Install

Best way to upload large amounts of data to servers/Cloud

Scheduled Pinned Locked Moved Discuss
7 Posts 6 Posters 182 Views
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • P Offline
    P Offline
    privsec
    wrote on last edited by
    #1

    Hello,
    I have been smacking my head against a wall for the past week. I have 12 TB of data that I need uploaded to two different targets.
    Im at present working with a residential upload connection of 20mbps max upload (this will take forever, I know) but my issue is I am using cyberduck and I have connections via idrive and my server set up via SSH, however, cyberduck seems to be always having a fit after I attempt to upload more then .01 kbs (Im not kidding).

    What have you done to implement a proper backup solution?

    robiR fbartelsF 2 Replies Last reply
    1
  • robiR Offline
    robiR Offline
    robi
    replied to privsec on last edited by
    #2

    @privsec

    1. visit a friend/place who has a blazingly fast connection. (corps, ISPs, datacenters, universities, .FTTH, ..)
    2. avoid uploading twice. (upload once, then xfer from there to the other)

    There are many clients that can access S3, so you can try those to see if they work better or use a different configuration.

    Avoid large transfers over SSH as it's inherently slow.

    Life of sky tech

    1 Reply Last reply
    1
  • fbartelsF Offline
    fbartelsF Offline
    fbartels App Dev
    replied to privsec on last edited by
    #3

    @privsec https://rclone.org/ is a good tool to interact with different cloud apis, but in itself it is not a backup tool.

    1 Reply Last reply
    2
  • G Offline
    G Offline
    GregY
    wrote on last edited by
    #4

    FWIW, and I'm far from an expert, but I use Backblaze B2 for my backups. I previously used Wasabi but I switched to Backblaze B2 since they can mail you a physical drive with your data on it should you need to restore from backup.

    I have about 10 TB total on my NAS that I need saving and, so far, I've backed up ~9 TB of it over the last 6 or so months As fbartels suggested, rclone is a good tool for this.

    I simply SSH into my QNAP NAS, and use 'rclone sync' to upload more data. Since I also have automatic local backups - an older NAS which is located in another room from my primary NAS - I don't need to upload everything all at once to BBB2. I normally upload 30G or so per day. Once I have the bulk of it done, I'll write a bash script for my NAS that will run in the middle of the night to automate my backups.

    Using rclone takes some trial and error to make sure you're uploading what you want, when you want. Luckily there's a --dry-run flag that will simulate your backup without actually modifying files or uploading any data.

    I also use Backblaze B2 as my cloudron backup option.

    1 Reply Last reply
    5
  • P Offline
    P Offline
    privsec
    wrote on last edited by
    #5

    Thanks all, Ill look into these options

    1 Reply Last reply
    0
  • marcusquinnM Offline
    marcusquinnM Offline
    marcusquinn
    wrote on last edited by
    #6
    • https://www.backblaze.com/b2/solutions/datatransfer/fireball.html

    Perhaps others offer similar. If you can get the storage temporarily onto something with a faster connection, that's probably your cheapest route.

    We're not here for a long time - but we are here for a good time :)
    Jersey/UK
    Work & Ecommerce Advice: https://brandlight.org
    Personal & Software Tips: https://marcusquinn.com

    jdaviescoatesJ 1 Reply Last reply
    1
  • jdaviescoatesJ Offline
    jdaviescoatesJ Offline
    jdaviescoates
    replied to marcusquinn on last edited by
    #7

    @marcusquinn said in Best way to upload large amounts of data to servers/Cloud:

    Perhaps others offer similar

    I'm pretty sure you used to be able to post Amazon DVDs or hard drives full of data, but not sure you still can and couldn't find anything about after a quick search

    I use Cloudron with Gandi & Hetzner

    1 Reply Last reply
    0

  • Login

  • Don't have an account? Register

  • Login or register to search.
  • First post
    Last post
0
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks