Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Feature Requests
  3. Speed up rsync backups to S3 Object Storage

Speed up rsync backups to S3 Object Storage

Scheduled Pinned Locked Moved Feature Requests
5 Posts 2 Posters 1.1k Views 2 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • robiR Offline
      robiR Offline
      robi
      wrote on last edited by robi
      #1

      Since rsync uses a lot of chatter to xfer small files one at a time, this is the most inefficient way to talk to an object store.

      Instead if there was a configurable "chunk" size of say 100MB saved locally first, then transferred via rsync, you'd get speed and efficiency gains.

      Conscious tech

      1 Reply Last reply
      2
      • girishG Offline
        girishG Offline
        girish
        Staff
        wrote on last edited by
        #2

        The S3 protocol is file based and not rsync based. We don't have a choice

        robiR 1 Reply Last reply
        1
        • girishG girish

          The S3 protocol is file based and not rsync based. We don't have a choice

          robiR Offline
          robiR Offline
          robi
          wrote on last edited by
          #3

          @girish you're not understanding this..

          we're not changing the protocol.

          we're simply pivoting what we point rsync at.
          not the app and all it's small files directly,
          but some collection of them neatly arranged in larger chunks files which all happens locally in a temp dir, before it's shuttled off to S3 efficiently.

          This should also save space on the multipart overhead issue, which seems like another feature to address.

          Conscious tech

          girishG 1 Reply Last reply
          0
          • robiR robi

            @girish you're not understanding this..

            we're not changing the protocol.

            we're simply pivoting what we point rsync at.
            not the app and all it's small files directly,
            but some collection of them neatly arranged in larger chunks files which all happens locally in a temp dir, before it's shuttled off to S3 efficiently.

            This should also save space on the multipart overhead issue, which seems like another feature to address.

            girishG Offline
            girishG Offline
            girish
            Staff
            wrote on last edited by
            #4

            @robi Ah ok, some sort of differential backup with "packing". I think duplicity does something like this (or maybe it's duplicati, i get confused between the two).

            1 Reply Last reply
            1
            • robiR Offline
              robiR Offline
              robi
              wrote on last edited by
              #5

              Where is this slated release wise?

              Conscious tech

              1 Reply Last reply
              1
              Reply
              • Reply as topic
              Log in to reply
              • Oldest to Newest
              • Newest to Oldest
              • Most Votes


                • Login

                • Don't have an account? Register

                • Login or register to search.
                • First post
                  Last post
                0
                • Categories
                • Recent
                • Tags
                • Popular
                • Bookmarks
                • Search