Backup Failures on Large Installations – MinIO Multipart-Upload Limit
-
Hello everyone,
I ran into an issue with a large Cloudron/Box installation where the backup aborts with this log entry:
… Upload progress: {"loaded":104878571520,"part":10002,…} BoxError: backuptask exited with code 1 reason: “Internal Error”Investigation
• Both AWS S3 and MinIO enforce a maximum of 10,000 parts per multipart upload.
• Default part size in Cloudron/Box is 10 MB → max transferable per upload ≈ 100 GB (10 000 × 10 MB).
• In my case part number 10 002 was attempted → limit exceeded → upload aborted.Part Size vs. Maximum Backup Size
Part Size Max Parts Max Data Volume 10 MB 10 000 100 GB 50 MB 10 000 500 GB 100 MB 10 000 1 000 GB 150 MB 10 000 1 500 GB Solution: Increase Part Size
This reduces the number of parts and allows backups up to ~500 GB without hitting the multipart limit.Sources
• AWS S3 Multipart-Upload Overview: https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
• MinIO Limits (10 000 parts, part size 5 MiB–5 TiB): https://github.com/minio/minio/blob/master/docs/minio-limits.mdGreetings
Christian -
Hello everyone,
I ran into an issue with a large Cloudron/Box installation where the backup aborts with this log entry:
… Upload progress: {"loaded":104878571520,"part":10002,…} BoxError: backuptask exited with code 1 reason: “Internal Error”Investigation
• Both AWS S3 and MinIO enforce a maximum of 10,000 parts per multipart upload.
• Default part size in Cloudron/Box is 10 MB → max transferable per upload ≈ 100 GB (10 000 × 10 MB).
• In my case part number 10 002 was attempted → limit exceeded → upload aborted.Part Size vs. Maximum Backup Size
Part Size Max Parts Max Data Volume 10 MB 10 000 100 GB 50 MB 10 000 500 GB 100 MB 10 000 1 000 GB 150 MB 10 000 1 500 GB Solution: Increase Part Size
This reduces the number of parts and allows backups up to ~500 GB without hitting the multipart limit.Sources
• AWS S3 Multipart-Upload Overview: https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
• MinIO Limits (10 000 parts, part size 5 MiB–5 TiB): https://github.com/minio/minio/blob/master/docs/minio-limits.mdGreetings
Christian@CBCUN said in Backup Failures on Large Installations – MinIO Multipart-Upload Limit:
Run in your Cloudron/Box environment:
export CLOUDRON_BACKUP_S3_PART_SIZE=50mb
Could you follow up on that with a bit more explanation?
Run in your Cloudron/Box environmentso ssh into the server and as root justexport CLOUDRON_BACKUP_S3_PART_SIZE=50mb?
This sounds wrong
So could you please provide a more in depth guide, that be awesome! -
N nebulon marked this topic as a question on
-
N nebulon has marked this topic as solved on
-
N nebulon has marked this topic as solved on
-
C CBCUN referenced this topic on
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login