Large app backup (~70GB) fails at rotate-copy step with `NoSuchKey` — IONOS S3 read-after-write consistency, but Cloudron has no retry

[dsp76]

Cloudron version: 10.0.4
Backup storage: IONOS S3-compatible Object Storage

Symptom:
The backup upload itself completes successfully, but the immediately following server-side rotate-copy (UploadPartCopy) fails with NoSuchKey, causing the whole backup task to fail with Old backup not found: snapshot/app_<id>.tar.gz.enc. Reproduced on two consecutive daily runs, always on our largest app (~70GB snapshot); other apps (4–70GB) in the same runs succeed.

Timeline (Task 11372, millisecond precision):

04:32:30.199  upload stats logged: 72,608,517,172 bytes transferred
04:32:30.285  backupupload: upload completed. error: null   (CompleteMultipartUpload -> 200 OK)
04:32:30.466  Copying (multipart) snapshot/app_c08ad55d-...tar.gz.enc   (rotate-copy starts, 181ms later)
04:32:30.534  Copying part 1 - ... bytes=0-1073741823
04:32:30.535  Copying part 2 - ... bytes=1073741824-2147483647
04:32:30.535  Copying part 3 - ... bytes=2147483648-3221225471
04:32:30.680  Aborting multipart copy   (145ms after part requests)
04:32:30.748  storage/s3: copy error: NoSuchKey: UnknownError
04:32:30.748  copy to .../app_registry.20zen.de_v1.13.0.tar.gz.enc errored. error: Old backup not found

Analysis:
CompleteMultipartUpload returns success at .285, but the immediately following UploadPartCopy on the exact same key returns NoSuchKey ~150ms later. This looks like a read-after-write consistency delay on the IONOS backend for very large multipart objects — AWS S3 itself has guaranteed strict read-after-write consistency (including for multipart uploads) since Dec 2020, so this is non-standard behavior on the storage backend side.

That said, Cloudron's rotate-copy step (s3.js: copyInternal) has no resilience for this: no retry, no backoff, no existence check before issuing the copy. A single transient 404 immediately fails the whole backup task, even though a short retry would very likely succeed.

Request: Could a short retry/backoff (or a HeadObject check before copying) be added around the rotate-copy step for large/multipart-uploaded objects, to tolerate brief backend propagation delays on non-AWS S3-compatible providers?

(I will also open a ticket at IONOS, but this little improvement would make Cloudron Backups more stable for similar situations with other providers too.)

Relevant stack:

BoxError: Old backup not found: snapshot/app_c08ad55d-….tar.gz.enc
    at throwError (file:///home/yellowtent/box/src/storage/s3.js:568:49)
    at copyInternal (file:///home/yellowtent/box/src/storage/s3.js:636:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
    at async Object.copy (file:///home/yellowtent/box/src/storage/s3.js:670:12)
    at async Object.copy (file:///home/yellowtent/box/src/backupformat/tgz.js:294:5)

related:
https://forum.cloudron.io/topic/14253/error-400-in-backup-process-with-ionos-s3-object-storage/38?_=1788503267379

[dsp76]

IONOS responded this:

"We are actively investigating the reported behavior. To support our Cloud Engineering team in conducting deeper log analysis within the storage cluster, we kindly request that you provide the following details:

• Target bucket name & IONOS S3 endpoint used during testing
• Request IDs (x-amz-request-id and x-amz-id-2 from the HTTP response headers) for both the CompleteMultipartUpload and the failed UploadPartCopy call
• Timestamps and timezone of the affected runs (or new, millisecond-precision logs if the error can be reproduced again)

While our Engineering team investigates the merging and propagation behavior for large multipart objects in the backend, we recommend testing one of the following client-side workarounds:

1. HeadObject polling with exponential backoff: Before executing UploadPartCopy on newly merged objects larger than 50 GB, insert a short polling loop using HeadObject (with retries on NoSuchKey / 404 Not Found), combined with exponential backoff and jitter. This ensures object metadata is globally visible across all gateway nodes before dependent copy operations are performed.

2. Insert a fixed delay: If implementing a polling mechanism in your Cloudron workflow is not straightforward, inserting a static delay of 1–2 seconds between CompleteMultipartUpload and subsequent copy actions can effectively prevent this race condition issue.

Please let us know whether applying one of these workarounds resolves the issue in your backup runs, and feel free to send us the requested logs once they are available."

I'm not sure if I can get the x-amz-request-id and x-amz-id-2 headers for them.
What do you think about their suggestions?

[james]

Hello @dsp76

@dsp76 said:

HeadObject polling with exponential backoff: Before executing UploadPartCopy on newly merged objects larger than 50 GB, insert a short polling loop using HeadObject (with retries on NoSuchKey / 404 Not Found), combined with exponential backoff and jitter. This ensures object metadata is globally visible across all gateway nodes before dependent copy operations are performed.

This is an interesting suggestion.
Will need to understand it more in depth tho.

[dsp76]

Hi @james I think it means that - once Cloudron receives an error like NoSuchKey oder 404 not found when polling the object directly after upload - it still retries a number of times with exponentially growing delays added with some random delays ("Jitter"). So the error might only be temporal.

[dsp76]

@james IONOS Support analyzed the situation and comes back with this:

Hello,

Thank you for your patience. Our Platform Engineering team has reviewed and analyzed the log files you provided. Based on this analysis, we can provide the following feedback:

1. Propagation delay: There is no artificial delay introduced by our platform. Objects become visible across different operations almost immediately, although not strictly in real time.
2. Object consistency: Once a CompleteMultipartUpload operation returns the HTTP status code 200 OK, subsequent HeadObject requests should succeed immediately. Shortly thereafter, all other operations should also become available.

Based on our log analysis, all multipart uploads were completed successfully. However, we observed subsequent UploadPartCopy requests referring to an UploadId that no longer exists, as the UploadId becomes invalid once an upload has been finalized.

Could you please clarify where these UploadPartCopy requests originate from within your application or client workflow? It appears that after completion of the original upload, an automated post-processing script or background process on your side is being executed, such as a CopyObject operation.

---

Could you please check now on cloudron side? Thanks a lot!

[james]

Hello @dsp76
I am currently investigating the issue and trying to reproduce it.
Did you do any changes to the backup site settings for the part size https://docs.cloudron.io/backups#part-size?
Please share with me the settings of your IONOS backup site.

[james]

Hello @dsp76
Can you also please provide me the full backup log of that failing backup?
I need to verify a pattern.

[james]

I have now measured the whole thing on IONOS with Cloudron 10.0.4, and the cause is clear.
It is on the IONOS side, but Cloudron hides the part that shows it.

Short version: on IONOS, overwriting (and deleting) an object costs about 24 seconds per GiB, handled synchronously before the response. Above roughly 25 GiB that exceeds the 600 s timeout of the proxy in front of their S3 service. The AWS SDK then silently repeats the CompleteMultipartUpload, one attempt answers 200 with an empty ETag, and afterwards the object is listed but its data can not be read. The rotate copy then fails with exactly the NoSuchKey from the first post.

@dsp76, three things would help:

1. In the log of a failed backup task: is there a long gap (around 10, 20 or 30 minutes) between the last "Uploading backup …" line and "upload completed"?
2. How large are the other apps' snapshots? By these measurements, the problem should only affect apps above roughly 25 GiB, which would explain why only your largest app fails.
3. Can the first MiB of the current snapshot still be downloaded, or does that return NoSuchKey? For example: aws s3api get-object --bucket <bucket> --key <prefix>/snapshot/app_c08ad55d-….tar.gz.enc --range bytes=0-1048575 --endpoint-url <your IONOS endpoint> /tmp/snapshot-test.bin

Below is a separate summary for IONOS support with all request IDs.

---

For IONOS support (all times UTC, 2026-09-11)

Hello IONOS team,

we have measured this with Cloudron's own S3 code and logged every request with x-amz-request-id. There are two separate issues: the runtime of an overwrite, and an object that is confirmed but unreadable afterwards.

Client and test setup

• Endpoint https://s3.eu-central-3.ionoscloud.com, region de, virtual-hosted-style requests, AWS SDK for JavaScript v3 (@aws-sdk/client-s3 3.1116.0), Cloudron 10.0.4, one client host
• Buckets repro-15914, repro-15914-b, repro-15914-c, repro-15914-d; objects of random data uploaded as multipart, 3 parts in parallel

1. CompleteMultipartUpload of an overwrite scales with the size of the object being replaced

Case	CompleteMultipartUpload
Key does not exist yet (create), 0.1 GiB to 39 GiB	under 1 s in every case
Key already exists (overwrite)	about 23.5 s per GiB of the old object
DeleteObject on such an object	about 25 s per GiB

Independent of the part size (50, 100 and 250 MiB parts at the same size give the same time) and of the bucket. Reproducible cheaply: create a 3.2 GiB object and overwrite it, which takes about 75 s instead of about 0.2 s. Occasionally, in 3 of about 34 operations, an overwrite or delete was fast (0.2–0.5 s).

Question: why is this work done synchronously before the response, and what happens with it after 600 s? A proxy in front of the service answers with 504 Gateway Time-out (HTML, no S3 headers, no request ID) after exactly 600.0 s. For comparison, AWS S3 sends the 200 header early for long-running completions and keeps the connection alive with whitespace. source: https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html

2. Above 600 s the object ends up unreadable

Bucket repro-15914-d, key repro-15914/snapshot/app_p2-s40000-r1.tar.gz.enc, 41,943,040,000 bytes in 800 parts, upload 2~C7vvA9XH1m0gX5ETzPptO6piVDoqJKw:

Sent	Request	Response	x-amz-request-id
22:16:20.100	CompleteMultipartUpload, attempt 1	504 after 600.07 s	none
22:26:40.176	CompleteMultipartUpload, attempt 2	504 after 600.06 s	none
22:37:00.241	CompleteMultipartUpload, attempt 3	200 in 78 ms, body contains <ETag></ETag>	tx00000898b3e6a0332a144-006aa4828c-810022444-eu-central-3
22:37:00.328	HeadObject	200, 41943040000 bytes, ETag "d6198b07dbf3a8ecd1f840d8b24c7c35-800", Last-Modified 22:26:40	tx0000019e93fb95908b459-006aa4828c-809736856-eu-central-3
22:37:00.478	UploadPartCopy part 1, bytes=0-1073741823	404 NoSuchKey, empty <Message>	tx000009e4ad2fa8b99db77-006aa4828c-809736856-eu-central-3
22:37:00.483	UploadPartCopy part 2	404 NoSuchKey	tx00000fe72582a902f5490-006aa4828c-807519665-eu-central-3
22:37:00.530	UploadPartCopy part 3	404 NoSuchUpload (our client had already aborted the destination upload)	tx00000d35fc6d29c7099a6-006aa4828c-809730471-eu-central-3
22:37:01.068	GET bytes=0-1048575	404 NoSuchKey	tx00000849e27e44423877a-006aa4828d-797521041-eu-central-3
22:37:01.110	GET bytes=20971520000-20972568575	404 NoSuchKey	tx000002052bd3daec1d0ba-006aa4828d-809730471-eu-central-3
22:37:01.197	GET bytes=41941991424-41943039999	206, 1048576 bytes	tx00000ad11ad4192c75825-006aa4828d-803482478-eu-central-3
22:40:31	the same three ranges again	404 / 404 / 206, unchanged	tx0000095e483f0c394dbf5-…, tx0000017020a5bb1e46bd8-…, tx0000018734af690b0297b-…

Control in the same bucket: repro-15914/repro/2026-09-11T21-50-54-233Z/app_repro-15914_v1.tar.gz.enc (39 GiB, created by a server-side copy) returns 206 for all three ranges at 22:40:31.

The same happened earlier in bucket repro-15914 with a 72.6 GB object: three 504s, then 200 with an empty ETag, and after that all three ranges returned NoSuchKey, still 33 minutes later. That object is still in the bucket for you to look at, key repro-15914/snapshot/app_repro-15914.tar.gz.enc, ETag "252d72dfd55e5f6b8799cb56a64339b3-1385", along with an intact control object.

Just to make sure this is not a timing issue and the large object might now readable:
All three ranges of the first manual object return NoSuchKey:

Range	Result	x-amz-request-id
bytes=0-1048575	404 NoSuchKey	tx00000760487e9eeddc8f0-006aa4a32b-809730471-eu-central-3
bytes=36304322560-36305371135	404 NoSuchKey	tx000008f860d7695a5e683-006aa4a32b-803482478-eu-central-3
bytes=72607596544-72608645119	404 NoSuchKey	tx0000037d2d2c104ca6b58-006aa4a32b-803482478-eu-central-3

HeadObject still claims the object is fine: HTTP 200, 72,608,645,120 bytes, ETag "252d72dfd55e5f6b8799cb56a64339b3-1385", Last-Modified 14:49:28, identical to 10 hours ago (tx0000095cf0eba4266169e-006aa4a32b-799478461-eu-central-3).

Questions

1. Why does CompleteMultipartUpload of an overwrite take about 24 s per GiB, and is the 504 produced by a proxy with a 600 s timeout?
2. Attempt 3 answered 200 with an empty ETag. Which request actually completed the upload? The object's Last-Modified matches attempt 2, which received a 504.
3. Why is the object listed and returned by HeadObject, while GET and UploadPartCopy answer NoSuchKey on several gateway hosts, unchanged minutes later? Is the data lost, and can it be recovered?
4. Is the empty ETag in the CompleteMultipartUpload response a reliable marker on your side that the object was not stored correctly?
5. You mentioned requests referring to an upload ID that no longer exists. In our logs the only upload IDs sent more than once belong to the repeated CompleteMultipartUpload attempts; the part 3 NoSuchUpload above comes from our own abort. Could those be the requests you found?

The affected objects are kept in repro-15914 and repro-15914-d.
If needed, I can provide the full log files of each attempt of reproducing this.