Backup Failures on Large Installations – MinIO Multipart-Upload Limit
-
Hello everyone,
I ran into an issue with a large Cloudron/Box installation where the backup aborts with this log entry:
… Upload progress: {"loaded":104878571520,"part":10002,…} BoxError: backuptask exited with code 1 reason: “Internal Error”
Investigation
• Both AWS S3 and MinIO enforce a maximum of 10,000 parts per multipart upload.
• Default part size in Cloudron/Box is 10 MB → max transferable per upload ≈ 100 GB (10 000 × 10 MB).
• In my case part number 10 002 was attempted → limit exceeded → upload aborted.Part Size vs. Maximum Backup Size
Part Size Max Parts Max Data Volume 10 MB 10 000 100 GB 50 MB 10 000 500 GB 100 MB 10 000 1 000 GB 150 MB 10 000 1 500 GB Solution: Increase Part Size
This reduces the number of parts and allows backups up to ~500 GB without hitting the multipart limit.Sources
• AWS S3 Multipart-Upload Overview: https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
• MinIO Limits (10 000 parts, part size 5 MiB–5 TiB): https://github.com/minio/minio/blob/master/docs/minio-limits.mdGreetings
Christian -
Hello everyone,
I ran into an issue with a large Cloudron/Box installation where the backup aborts with this log entry:
… Upload progress: {"loaded":104878571520,"part":10002,…} BoxError: backuptask exited with code 1 reason: “Internal Error”
Investigation
• Both AWS S3 and MinIO enforce a maximum of 10,000 parts per multipart upload.
• Default part size in Cloudron/Box is 10 MB → max transferable per upload ≈ 100 GB (10 000 × 10 MB).
• In my case part number 10 002 was attempted → limit exceeded → upload aborted.Part Size vs. Maximum Backup Size
Part Size Max Parts Max Data Volume 10 MB 10 000 100 GB 50 MB 10 000 500 GB 100 MB 10 000 1 000 GB 150 MB 10 000 1 500 GB Solution: Increase Part Size
This reduces the number of parts and allows backups up to ~500 GB without hitting the multipart limit.Sources
• AWS S3 Multipart-Upload Overview: https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
• MinIO Limits (10 000 parts, part size 5 MiB–5 TiB): https://github.com/minio/minio/blob/master/docs/minio-limits.mdGreetings
Christian@CBCUN said in Backup Failures on Large Installations – MinIO Multipart-Upload Limit:
Run in your Cloudron/Box environment:
export CLOUDRON_BACKUP_S3_PART_SIZE=50mb
Could you follow up on that with a bit more explanation?
Run in your Cloudron/Box environment
so ssh into the server and as root justexport CLOUDRON_BACKUP_S3_PART_SIZE=50mb
?
This sounds wrong
So could you please provide a more in depth guide, that be awesome!