Minio as Backup - Timeout/Slowdown

[doodlemania2]

Is there a suggested concurrency limit I should put on backups when leveraging Minio (currently just taking the defaults). Getting lots of these, and an eventual permafail.

SlowDown: A timeout exceeded while waiting to proceed with the request, please reduce your request rate 503

(Since using rsync, easy enough to just start it back up, but would prefer to not overwhelm it) - also looked at upping the startup args on the minio app to increase defaults, but not well documented on their side (says "unlimited")

[robi]

I think you're one of the first to do this, so keep bumping up the numbers and see how many less errors you get until you don't.

rsync will need a different setting than tgz but will also likely hold the upper bound.

[girish]

@doodlemania2 Do you see any errors in the minio logs? I haven't seen any SlowDown event from minio.

[doodlemania2]

@girish yeah, currently everything in Minio is set to default and backup config is also set to default (10 and can't go lower). I get lots of these:
Dec 21 09:32:32 box:tasks 1719: {"percent":27.923076923076927,"message":"Retrying (5) copy of data/files/.... Error: SlowDown: A timeout exceeded while waiting to proceed with the request, please reduce your request rate 503 (media.apps.thedoodleproject.net)"}

After a while, the timeouts go to Retrying (10) ... and the whole job goes boom. Restart picks up where it left off thankfully, but yeah.

I dug around Minio configs - it looks "unlimited" by default, so I wonder if it is calculating 'limits' based on memory available - so I could turn RAM up or something.

[girish]

@doodlemania2 Maybe we can ask on the upstream bug tracker as to what might cause this. I tried searching there but didn't find anything interesting.

[doodlemania2]

@girish on it!

[doodlemania2]

@doodlemania2 https://github.com/minio/minio/issues/11147

[doodlemania2]

@doodlemania2 They sent me here: https://github.com/minio/minio/tree/master/docs/config#api which is good for explaining how to adjust it, but was more hoping for an understanding of how it calculates it based on server resources. Suffice to say, I bumped it up to 8GB of RAM and the errors have vanished!

[robi]

@doodlemania2 whoa.. what was the RAM setting before?

Sounds like it needs to release some memory more regularly before it's done with a long backup via rsync.

[doodlemania2]

@robi the app defaults - 256mb

[robi]

@doodlemania2 can you test it with other increments to see where the threshold is?

[doodlemania2]

@robi Set it to 4GB and it slowed way down, so, I'd say anything below that would be too little.

[robi]

@doodlemania2 can you report your findings to the Minio team and see if they have a solution?

[doodlemania2]

@robi I did - they recommended nothing less than 4GB for anything "production" worthy. As Cloudron's default is 256MB, that surely splains most of it.

[robi]

@doodlemania2 I hear you.

The question was more about optimizing that requirement and more frequent checkpoints or memory free events to not require an ever rising memory requirement of such large swings.

[doodlemania2]

@robi will inquire!