Large app backup (~70GB) fails at rotate-copy step with `NoSuchKey` — IONOS S3 read-after-write consistency, but Cloudron has no retry

[dsp76]

Cloudron version: 10.0.4
Backup storage: IONOS S3-compatible Object Storage

Symptom:
The backup upload itself completes successfully, but the immediately following server-side rotate-copy (UploadPartCopy) fails with NoSuchKey, causing the whole backup task to fail with Old backup not found: snapshot/app_<id>.tar.gz.enc. Reproduced on two consecutive daily runs, always on our largest app (~70GB snapshot); other apps (4–70GB) in the same runs succeed.

Timeline (Task 11372, millisecond precision):

04:32:30.199  upload stats logged: 72,608,517,172 bytes transferred
04:32:30.285  backupupload: upload completed. error: null   (CompleteMultipartUpload -> 200 OK)
04:32:30.466  Copying (multipart) snapshot/app_c08ad55d-...tar.gz.enc   (rotate-copy starts, 181ms later)
04:32:30.534  Copying part 1 - ... bytes=0-1073741823
04:32:30.535  Copying part 2 - ... bytes=1073741824-2147483647
04:32:30.535  Copying part 3 - ... bytes=2147483648-3221225471
04:32:30.680  Aborting multipart copy   (145ms after part requests)
04:32:30.748  storage/s3: copy error: NoSuchKey: UnknownError
04:32:30.748  copy to .../app_registry.20zen.de_v1.13.0.tar.gz.enc errored. error: Old backup not found

Analysis:
CompleteMultipartUpload returns success at .285, but the immediately following UploadPartCopy on the exact same key returns NoSuchKey ~150ms later. This looks like a read-after-write consistency delay on the IONOS backend for very large multipart objects — AWS S3 itself has guaranteed strict read-after-write consistency (including for multipart uploads) since Dec 2020, so this is non-standard behavior on the storage backend side.

That said, Cloudron's rotate-copy step (s3.js: copyInternal) has no resilience for this: no retry, no backoff, no existence check before issuing the copy. A single transient 404 immediately fails the whole backup task, even though a short retry would very likely succeed.

Request: Could a short retry/backoff (or a HeadObject check before copying) be added around the rotate-copy step for large/multipart-uploaded objects, to tolerate brief backend propagation delays on non-AWS S3-compatible providers?

(I will also open a ticket at IONOS, but this little improvement would make Cloudron Backups more stable for similar situations with other providers too.)

Relevant stack:

BoxError: Old backup not found: snapshot/app_c08ad55d-2629-4de5-9d8c-adc28c142aca.tar.gz.enc
    at throwError (file:///home/yellowtent/box/src/storage/s3.js:568:49)
    at copyInternal (file:///home/yellowtent/box/src/storage/s3.js:636:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
    at async Object.copy (file:///home/yellowtent/box/src/storage/s3.js:670:12)
    at async Object.copy (file:///home/yellowtent/box/src/backupformat/tgz.js:294:5)

related:
https://forum.cloudron.io/topic/14253/error-400-in-backup-process-with-ionos-s3-object-storage/38?_=1788503267379