Minio as Backup - Timeout/Slowdown

[robi]

I think you're one of the first to do this, so keep bumping up the numbers and see how many less errors you get until you don't.

rsync will need a different setting than tgz but will also likely hold the upper bound.

[girish]

@doodlemania2 Do you see any errors in the minio logs? I haven't seen any SlowDown event from minio.

[doodlemania2]

@girish yeah, currently everything in Minio is set to default and backup config is also set to default (10 and can't go lower). I get lots of these:
Dec 21 09:32:32 box:tasks 1719: {"percent":27.923076923076927,"message":"Retrying (5) copy of data/files/.... Error: SlowDown: A timeout exceeded while waiting to proceed with the request, please reduce your request rate 503 (media.apps.thedoodleproject.net)"}

After a while, the timeouts go to Retrying (10) ... and the whole job goes boom. Restart picks up where it left off thankfully, but yeah.

I dug around Minio configs - it looks "unlimited" by default, so I wonder if it is calculating 'limits' based on memory available - so I could turn RAM up or something.

[girish]

@doodlemania2 Maybe we can ask on the upstream bug tracker as to what might cause this. I tried searching there but didn't find anything interesting.

[doodlemania2]

@girish on it!

[doodlemania2]

@doodlemania2 https://github.com/minio/minio/issues/11147

[doodlemania2]

@doodlemania2 They sent me here: https://github.com/minio/minio/tree/master/docs/config#api which is good for explaining how to adjust it, but was more hoping for an understanding of how it calculates it based on server resources. Suffice to say, I bumped it up to 8GB of RAM and the errors have vanished!

[robi]

@doodlemania2 whoa.. what was the RAM setting before?

Sounds like it needs to release some memory more regularly before it's done with a long backup via rsync.

[doodlemania2]

@robi the app defaults - 256mb

[robi]

@doodlemania2 can you test it with other increments to see where the threshold is?

[doodlemania2]

@robi Set it to 4GB and it slowed way down, so, I'd say anything below that would be too little.

[robi]

@doodlemania2 can you report your findings to the Minio team and see if they have a solution?

[doodlemania2]

@robi I did - they recommended nothing less than 4GB for anything "production" worthy. As Cloudron's default is 256MB, that surely splains most of it.

[robi]

@doodlemania2 I hear you.

The question was more about optimizing that requirement and more frequent checkpoints or memory free events to not require an ever rising memory requirement of such large swings.

[doodlemania2]

@robi will inquire!

[Gaurav]

@doodlemania2 hey I am facing the same issue can you suggest to me where I can set the ram in Minio

[Gaurav]

@doodlemania2
hey I am facing the same issue can you suggest to me where I can set the ram in Minio

[robi]

@Gaurav it's not in minio, it's in Cloudron App Config for Minio.

[girish]

@Gaurav https://docs.cloudron.io/apps/#memory-limit