I have reproduced the issue with Cloudron 10.0.4 against IONOS S3 (Berlin, eu-central-3).
Short version: the failure is caused on the IONOS side, but Cloudron currently hides the part that shows it.
Setup
A small Node script calls Cloudron's own src/storage/s3.js upload() and copy() directly, i.e. the same code path a tgz backup uses: upload snapshot/app_<id>.tar.gz.enc, then the rotate copy to the versioned name.
Except for run 1, all runs were done from the same Cloudron 10.0.4 server, using its installed box code. The logs of runs 3, 4, A and B show that the AWS SDK packages are identical to the 10.0.4 package-lock.json. Nothing else wrote to the test buckets.
72.6 GB of random data (practically the same size as the snapshot in the first post), default upload part size 50 MiB = 1385 parts. The copy uses 1 GiB ranges, 3 in parallel, exactly like copyInternal().
Runs 3, 4, A and B log every S3 request and response with UTC timestamps, headers, body and x-amz-request-id. Runs 1 and 2 used an earlier version of the script with less detail.
Results
Bucket
Run
What
CompleteMultipartUpload
Rotate copy / read
1
run 1
create from my home connection, cancelled manually after at least 100 of 1385 parts
never sent; the unfinished upload stayed open
–
1
create on the server (log not saved)
create
no timeout (the whole upload took 12 min 19 s)
copy OK
1
run 2
overwrite
504 after 600 s, twice (run stopped while the 3rd attempt was waiting)
–
1
run 3
overwrite
504 after 600 s, three times, then 200
NoSuchKey → "Old backup not found"
1
run 4
copy the same snapshot again 30 min later
–
NoSuchKey
1
run 5
ranged GET of the snapshot
–
NoSuchKey, while HeadObject returns 200
1
run 6
ranged GET of the copy made by the server create run (control)
–
readable
2 (new)
run A
create
200 after 0.2 s
copy OK
2 (new)
run B
overwrite
200 after 0.5 s
copy OK
2 (new)
run C
ranged GET of the overwritten snapshot
–
readable
What happens in the failing case (run 3)
All 1385 parts upload without errors, then the AWS SDK sends CompleteMultipartUpload.
IONOS does not answer within 10 minutes. After exactly 600 s the request ends with 504 Gateway Time-out: an HTML page without S3 headers or request ID, so it looks like it comes from a proxy in front of the storage.
The AWS SDK treats 504 as a transient error and silently sends the same CompleteMultipartUpload again after 20 s. Cloudron logs nothing about this; the backup task just seems to hang at the end of the upload.
After three 504s (31 minutes), the 4th attempt returns 200 within 80 ms, but with an empty ETag. The upload counts as successful (in a real backup Cloudron logs "upload completed. error: null" here) and the rotate copy starts immediately.
HeadObject returns 200 with the full size and a new ETag, but every UploadPartCopy returns 404 NoSuchKey with an empty message. That is exactly the NoSuchKey: UnknownError / "Old backup not found" from the first post.
The object stays like that: 30 minutes later the copy fails again, and a plain ranged GET of the first, middle and last MiB also returns NoSuchKey. HeadObject and the bucket listing still show the object, while the copy made by the earlier create run in the same bucket reads fine with the same client.
So after this sequence the snapshot object is listed, but its data can not be read. Retrying the copy, or polling HeadObject before the copy as IONOS suggested, would not help: HeadObject returned 200 right before every failed read.
It does not happen every time
In a new bucket, a create followed by an overwrite of the same 72.6 GB object worked without problems: same server, same script, and nothing else writing to either bucket. CompleteMultipartUpload answered in 0.2 s and 0.5 s there, instead of timing out after 600 s three times. The differences I can see between the failing and the working overwrites:
In bucket 1, the upload I cancelled in run 1 was still open: an unfinished multipart upload for the same snapshot key, started 12:54 UTC and never completed or aborted. The create on the server also ran while it was open and was fine; only the overwrites hung. Bucket 2 had no unfinished uploads.
Time of day: the failing overwrites ran between 13:35 and 15:00 UTC, the working ones between 16:02 and 16:46 UTC.
Run 3 overwrote the object from run 2, whose own CompleteMultipartUpload had timed out. Run 2 however overwrote a normally created object and still hung.
I have not confirmed the unfinished upload as the trigger yet.
@dsp76 could you check three things?
In the log of a failed backup task: is there a long gap (around 10, 20 or 30 minutes) between the last "Uploading backup …" line and "upload completed"?
Are there unfinished multipart uploads in your bucket? For example with the AWS CLI: aws s3api list-multipart-uploads --bucket <bucket> --prefix <prefix>/snapshot/ --endpoint-url <your IONOS endpoint> (please don't abort anything while a backup is running).
Can the first MiB of the current snapshot still be downloaded, or does that return NoSuchKey as well? For example: aws s3api get-object --bucket <bucket> --key <prefix>/snapshot/app_c08ad55d-….tar.gz.enc --range bytes=0-1048575 --endpoint-url <your IONOS endpoint> /tmp/snapshot-test.bin
On the Cloudron side
A few things I noticed while going through this. Nothing is decided yet:
The 504 retries of CompleteMultipartUpload are invisible in the backup log. Logging SDK retries would have made this much easier to spot.
The SDK request header shows max=3, but that is only the SDK's default for the header. RETRY_STRATEGY in s3.js allows 10 attempts, and a 4th attempt did happen.
In copyInternal() the abort is sent while the other parallel part copies are still running (async.eachOfLimit does not cancel them). That can cause confusing follow-up errors in provider logs.
cleanup() only aborts unfinished multipart uploads older than 3 days, so an interrupted backup leaves one behind on the snapshot key for that long. Whether that matters depends on the question above.
Since the object is unreadable, failing the backup is the right outcome. A clearer error would help, e.g. a small ranged GET right after the upload to report "storage accepted the upload, but the object is not readable" instead of "Old backup not found".
I will attempt to reproduce this again with the manually cancelled upload to see if that caused the issue.
