Cloudron version: 10.0.4
Backup storage: IONOS S3-compatible Object Storage
Symptom:
The backup upload itself completes successfully, but the immediately following server-side rotate-copy (UploadPartCopy) fails with NoSuchKey, causing the whole backup task to fail with Old backup not found: snapshot/app_<id>.tar.gz.enc. Reproduced on two consecutive daily runs, always on our largest app (~70GB snapshot); other apps (4–70GB) in the same runs succeed.
Timeline (Task 11372, millisecond precision):
04:32:30.199 upload stats logged: 72,608,517,172 bytes transferred
04:32:30.285 backupupload: upload completed. error: null (CompleteMultipartUpload -> 200 OK)
04:32:30.466 Copying (multipart) snapshot/app_c08ad55d-...tar.gz.enc (rotate-copy starts, 181ms later)
04:32:30.534 Copying part 1 - ... bytes=0-1073741823
04:32:30.535 Copying part 2 - ... bytes=1073741824-2147483647
04:32:30.535 Copying part 3 - ... bytes=2147483648-3221225471
04:32:30.680 Aborting multipart copy (145ms after part requests)
04:32:30.748 storage/s3: copy error: NoSuchKey: UnknownError
04:32:30.748 copy to .../app_registry.20zen.de_v1.13.0.tar.gz.enc errored. error: Old backup not found
Analysis:
CompleteMultipartUpload returns success at .285, but the immediately following UploadPartCopy on the exact same key returns NoSuchKey ~150ms later. This looks like a read-after-write consistency delay on the IONOS backend for very large multipart objects — AWS S3 itself has guaranteed strict read-after-write consistency (including for multipart uploads) since Dec 2020, so this is non-standard behavior on the storage backend side.
That said, Cloudron's rotate-copy step (s3.js: copyInternal) has no resilience for this: no retry, no backoff, no existence check before issuing the copy. A single transient 404 immediately fails the whole backup task, even though a short retry would very likely succeed.
Request: Could a short retry/backoff (or a HeadObject check before copying) be added around the rotate-copy step for large/multipart-uploaded objects, to tolerate brief backend propagation delays on non-AWS S3-compatible providers?
(I will also open a ticket at IONOS, but this little improvement would make Cloudron Backups more stable for similar situations with other providers too.)
Relevant stack:
BoxError: Old backup not found: snapshot/app_c08ad55d-2629-4de5-9d8c-adc28c142aca.tar.gz.enc
at throwError (file:///home/yellowtent/box/src/storage/s3.js:568:49)
at copyInternal (file:///home/yellowtent/box/src/storage/s3.js:636:16)
at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
at async Object.copy (file:///home/yellowtent/box/src/storage/s3.js:670:12)
at async Object.copy (file:///home/yellowtent/box/src/backupformat/tgz.js:294:5)
related:
https://forum.cloudron.io/topic/14253/error-400-in-backup-process-with-ionos-s3-object-storage/38?_=1788503267379