Speed up rsync backups to S3 Object Storage
-
Since rsync uses a lot of chatter to xfer small files one at a time, this is the most inefficient way to talk to an object store.
Instead if there was a configurable "chunk" size of say 100MB saved locally first, then transferred via rsync, you'd get speed and efficiency gains.
-
@girish you're not understanding this..
we're not changing the protocol.
we're simply pivoting what we point rsync at.
not the app and all it's small files directly,
but some collection of them neatly arranged in larger chunks files which all happens locally in a temp dir, before it's shuttled off to S3 efficiently.This should also save space on the multipart overhead issue, which seems like another feature to address.
-
@girish you're not understanding this..
we're not changing the protocol.
we're simply pivoting what we point rsync at.
not the app and all it's small files directly,
but some collection of them neatly arranged in larger chunks files which all happens locally in a temp dir, before it's shuttled off to S3 efficiently.This should also save space on the multipart overhead issue, which seems like another feature to address.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login