I have now measured the whole thing on IONOS with Cloudron 10.0.4, and the cause is clear.
It is on the IONOS side, but Cloudron hides the part that shows it.
Short version: on IONOS, overwriting (and deleting) an object costs about 24 seconds per GiB, handled synchronously before the response. Above roughly 25 GiB that exceeds the 600 s timeout of the proxy in front of their S3 service. The AWS SDK then silently repeats the CompleteMultipartUpload, one attempt answers 200 with an empty ETag, and afterwards the object is listed but its data can not be read. The rotate copy then fails with exactly the NoSuchKey from the first post.
@dsp76, three things would help:
- In the log of a failed backup task: is there a long gap (around 10, 20 or 30 minutes) between the last "Uploading backup …" line and "upload completed"?
- How large are the other apps' snapshots? By these measurements, the problem should only affect apps above roughly 25 GiB, which would explain why only your largest app fails.
- Can the first MiB of the current snapshot still be downloaded, or does that return
NoSuchKey? For example:aws s3api get-object --bucket <bucket> --key <prefix>/snapshot/app_c08ad55d-….tar.gz.enc --range bytes=0-1048575 --endpoint-url <your IONOS endpoint> /tmp/snapshot-test.bin
Below is a separate summary for IONOS support with all request IDs.
For IONOS support (all times UTC, 2026-09-11)
Hello IONOS team,
we have measured this with Cloudron's own S3 code and logged every request with x-amz-request-id. There are two separate issues: the runtime of an overwrite, and an object that is confirmed but unreadable afterwards.
Client and test setup
- Endpoint
https://s3.eu-central-3.ionoscloud.com, regionde, virtual-hosted-style requests, AWS SDK for JavaScript v3 (@aws-sdk/client-s33.1116.0), Cloudron 10.0.4, one client host - Buckets
repro-15914,repro-15914-b,repro-15914-c,repro-15914-d; objects of random data uploaded as multipart, 3 parts in parallel
1. CompleteMultipartUpload of an overwrite scales with the size of the object being replaced
| Case | CompleteMultipartUpload |
|---|---|
| Key does not exist yet (create), 0.1 GiB to 39 GiB | under 1 s in every case |
| Key already exists (overwrite) | about 23.5 s per GiB of the old object |
| DeleteObject on such an object | about 25 s per GiB |
Independent of the part size (50, 100 and 250 MiB parts at the same size give the same time) and of the bucket. Reproducible cheaply: create a 3.2 GiB object and overwrite it, which takes about 75 s instead of about 0.2 s. Occasionally, in 3 of about 34 operations, an overwrite or delete was fast (0.2–0.5 s).
Question: why is this work done synchronously before the response, and what happens with it after 600 s? A proxy in front of the service answers with 504 Gateway Time-out (HTML, no S3 headers, no request ID) after exactly 600.0 s. For comparison, AWS S3 sends the 200 header early for long-running completions and keeps the connection alive with whitespace. source: https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html
2. Above 600 s the object ends up unreadable
Bucket repro-15914-d, key repro-15914/snapshot/app_p2-s40000-r1.tar.gz.enc, 41,943,040,000 bytes in 800 parts, upload 2~C7vvA9XH1m0gX5ETzPptO6piVDoqJKw:
| Sent | Request | Response | x-amz-request-id |
|---|---|---|---|
| 22:16:20.100 | CompleteMultipartUpload, attempt 1 | 504 after 600.07 s | none |
| 22:26:40.176 | CompleteMultipartUpload, attempt 2 | 504 after 600.06 s | none |
| 22:37:00.241 | CompleteMultipartUpload, attempt 3 | 200 in 78 ms, body contains <ETag></ETag> |
tx00000898b3e6a0332a144-006aa4828c-810022444-eu-central-3 |
| 22:37:00.328 | HeadObject | 200, 41943040000 bytes, ETag "d6198b07dbf3a8ecd1f840d8b24c7c35-800", Last-Modified 22:26:40 |
tx0000019e93fb95908b459-006aa4828c-809736856-eu-central-3 |
| 22:37:00.478 | UploadPartCopy part 1, bytes=0-1073741823 |
404 NoSuchKey, empty <Message> |
tx000009e4ad2fa8b99db77-006aa4828c-809736856-eu-central-3 |
| 22:37:00.483 | UploadPartCopy part 2 | 404 NoSuchKey | tx00000fe72582a902f5490-006aa4828c-807519665-eu-central-3 |
| 22:37:00.530 | UploadPartCopy part 3 | 404 NoSuchUpload (our client had already aborted the destination upload) | tx00000d35fc6d29c7099a6-006aa4828c-809730471-eu-central-3 |
| 22:37:01.068 | GET bytes=0-1048575 |
404 NoSuchKey | tx00000849e27e44423877a-006aa4828d-797521041-eu-central-3 |
| 22:37:01.110 | GET bytes=20971520000-20972568575 |
404 NoSuchKey | tx000002052bd3daec1d0ba-006aa4828d-809730471-eu-central-3 |
| 22:37:01.197 | GET bytes=41941991424-41943039999 |
206, 1048576 bytes | tx00000ad11ad4192c75825-006aa4828d-803482478-eu-central-3 |
| 22:40:31 | the same three ranges again | 404 / 404 / 206, unchanged | tx0000095e483f0c394dbf5-…, tx0000017020a5bb1e46bd8-…, tx0000018734af690b0297b-… |
Control in the same bucket: repro-15914/repro/2026-09-11T21-50-54-233Z/app_repro-15914_v1.tar.gz.enc (39 GiB, created by a server-side copy) returns 206 for all three ranges at 22:40:31.
The same happened earlier in bucket repro-15914 with a 72.6 GB object: three 504s, then 200 with an empty ETag, and after that all three ranges returned NoSuchKey, still 33 minutes later. That object is still in the bucket for you to look at, key repro-15914/snapshot/app_repro-15914.tar.gz.enc, ETag "252d72dfd55e5f6b8799cb56a64339b3-1385", along with an intact control object.
Just to make sure this is not a timing issue and the large object might now readable:
All three ranges of the first manual object return NoSuchKey:
| Range | Result | x-amz-request-id |
|---|---|---|
| bytes=0-1048575 | 404 NoSuchKey | tx00000760487e9eeddc8f0-006aa4a32b-809730471-eu-central-3 |
| bytes=36304322560-36305371135 | 404 NoSuchKey | tx000008f860d7695a5e683-006aa4a32b-803482478-eu-central-3 |
| bytes=72607596544-72608645119 | 404 NoSuchKey | tx0000037d2d2c104ca6b58-006aa4a32b-803482478-eu-central-3 |
HeadObject still claims the object is fine: HTTP 200, 72,608,645,120 bytes, ETag "252d72dfd55e5f6b8799cb56a64339b3-1385", Last-Modified 14:49:28, identical to 10 hours ago (tx0000095cf0eba4266169e-006aa4a32b-799478461-eu-central-3).
Questions
- Why does
CompleteMultipartUploadof an overwrite take about 24 s per GiB, and is the 504 produced by a proxy with a 600 s timeout? - Attempt 3 answered 200 with an empty ETag. Which request actually completed the upload? The object's Last-Modified matches attempt 2, which received a 504.
- Why is the object listed and returned by HeadObject, while GET and UploadPartCopy answer
NoSuchKeyon several gateway hosts, unchanged minutes later? Is the data lost, and can it be recovered? - Is the empty ETag in the CompleteMultipartUpload response a reliable marker on your side that the object was not stored correctly?
- You mentioned requests referring to an upload ID that no longer exists. In our logs the only upload IDs sent more than once belong to the repeated CompleteMultipartUpload attempts; the part 3
NoSuchUploadabove comes from our own abort. Could those be the requests you found?
The affected objects are kept in repro-15914 and repro-15914-d.
If needed, I can provide the full log files of each attempt of reproducing this.
