Error 400 in backup process with Ionos S3 Object Storage
-
Hello @dsp76
- Have you tested with alternative storage providers or alternative regions to isolate if this is Ionos S3/Zone-specific?
- Which region are you using for your S3 bucket?
- Where data origin is stored?
- How frequently does this issue occur?
-
Hi,
1.) we used Hetzner Storage Box before pretty stable. But thats a while ago. We needed to switch, as we didn't want our backup location to be the same provider as our server.2.) Region of IONOS S3 is: eu-central-3 (Berlin)
3.) Not sure what this means?
4.) Just now, not the days before. -
Hi,
1.) we used Hetzner Storage Box before pretty stable. But thats a while ago. We needed to switch, as we didn't want our backup location to be the same provider as our server.2.) Region of IONOS S3 is: eu-central-3 (Berlin)
3.) Not sure what this means?
4.) Just now, not the days before.- I was intending which was origin of data, in which data center (EU, US, and so on) is stored source, if far away from Ionos S3 Region (destination).
What I have in mind, it could be some link issues between source and destination...
Il I'm not wrong Hetzner Storage Box service does not have any S3 endpoint, .... Just sFtp, webdav, and others.
So what I can suggest to you is to debug using other alternative S3 service. You could try backblaze, idrive e2, or Exoscale.
Can you tell also us more about Cloudron instance resources? How much Ram do you have? Cpu? is Bare metal server or Vps?
I think all those informations help to better understand.
-
If we have a way to reliably reproduce the issue, we can see if there is a workaround for that S3 provider, but most likely it is some issue on provider end, if this is not happening all the time. Sadly the S3 implementations between provider are not always behaving the same way. We use the AWS S3 SDK for all requests, which is the standard.
-
Hi there,
its a Virtual Dedicated Server at Hetzner. Decently sized with 32GB RAM and 8 Cores.
When I drop the error message into AI, it also turns towards the provider. Suddenly HTTP400 when uploading ...
The concerned App ID is the largest - our own docker registry with about 40GB.https://docs.ionos.com/cloud/storage-and-backup/ionos-object-storage/overview/limitations
Currently I load it in multi upload parts of max. 512MB size. The backup process may use up to 10GB memory, which should be enough. For testing I reduced the size zu 128MB for the parts. Lets see if thats fixing it.
I also opened a ticket with IONOS to find out, what the cause for error 400 was at the given timestamp. Hope they find some helpful information in the logs.
-
-
Sorry, it still happens. Even reducing the chunk sizes to 128MB didn't solve it. For whatever reason, the backup task crashed on Cloudron due to "internal error". See the last lines...
What else could be the reason? Can I help to identify the root cause?
Sep 05 03:34:32 box:backupformat/tgz addToPack: added ./data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256] directory Sep 05 03:34:32 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/[UUID]/data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256] Sep 05 03:34:32 box:backupformat/tgz addToPack: added ./data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256]/data file Sep 05 03:34:32 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/[UUID]/data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256] Sep 05 03:34:32 box:backupformat/tgz addToPack: added ./data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256]/data file Sep 05 03:34:32 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/[UUID]/data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256] Sep 05 03:34:32 box:backupformat/tgz addToPack: added ./data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256]/data file Sep 05 03:34:32 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/[UUID]/data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256] Sep 05 03:34:32 box:backupformat/tgz addToPack: added ./data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256]/data file Sep 05 03:34:32 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/[UUID]/data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256] Sep 05 03:34:32 box:backupformat/tgz addToPack: added ./data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256]/data file Sep 05 03:34:32 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/[UUID]/data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256] Sep 05 03:34:32 box:backupformat/tgz addToPack: added ./data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256]/data file Sep 05 03:34:32 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/[UUID]/data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256] Sep 05 03:34:32 box:backupformat/tgz addToPack: added ./data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256]/data file Sep 05 03:34:32 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/[UUID]/data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256] Sep 05 03:34:32 box:backupformat/tgz addToPack: added ./data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256]/data file Sep 05 03:34:32 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/[UUID]/data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256] Sep 05 03:34:34 box:tasks update 7013: {"percent":77.66666666666667,"message":"Uploading backup 4803M@15MBps ([REDACTED])"} Sep 05 03:34:38 box:backupformat/tgz addToPack: added ./data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256]/data file Sep 05 03:34:38 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/[UUID]/data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256] Sep 05 03:34:38 box:backupformat/tgz addToPack: added ./data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256]/data file Sep 05 03:34:38 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/[UUID]/data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256] Sep 05 03:34:38 box:backupformat/tgz addToPack: added ./data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256]/data file Sep 05 03:34:38 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/[UUID]/data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256] Sep 05 03:34:38 box:backupformat/tgz addToPack: added ./data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256]/data file Sep 05 03:34:38 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/[UUID]/data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256] Sep 05 03:34:38 box:backupformat/tgz addToPack: added ./data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256]/data file Sep 05 03:34:38 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/[UUID]/data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256] Sep 05 03:34:38 box:backupformat/tgz addToPack: added ./data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256]/data file Sep 05 03:34:38 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/[UUID]/data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256] Sep 05 03:34:38 box:backupformat/tgz addToPack: added ./data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256]/data file Sep 05 03:34:38 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/[UUID]/data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256] Sep 05 03:34:38 box:backupformat/tgz addToPack: added ./data/storage/docker/registry/v2/blobs/sha256/[XX]/[SHA256]/data file Sep 05 03:34:39 box:storage/s3 Upload progress: {"loaded":4966055936,"part":38,"key":"cloudron_tz/snapshot/app_[UUID].tar.gz.enc"} Sep 05 03:34:39 /home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:712 Sep 05 03:34:39 resp.error = AWS.util.error(new Error(), { Sep 05 03:34:39 ^ Sep 05 03:34:39 Sep 05 03:34:39 400: null Sep 05 03:34:39 at Request.extractError (/home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:712:35) Sep 05 03:34:39 at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20) Sep 05 03:34:39 at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10) Sep 05 03:34:39 at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:686:14) Sep 05 03:34:39 at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10) Sep 05 03:34:39 at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12) Sep 05 03:34:39 at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10 Sep 05 03:34:39 at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9) Sep 05 03:34:39 at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:688:12) Sep 05 03:34:39 at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:116:18) { Sep 05 03:34:39 code: 400, Sep 05 03:34:39 region: null, Sep 05 03:34:39 time: 2025-09-05T01:34:41.375Z, Sep 05 03:34:39 requestId: null, Sep 05 03:34:39 extendedRequestId: undefined, Sep 05 03:34:39 cfId: undefined, Sep 05 03:34:39 statusCode: 400, Sep 05 03:34:39 retryable: false, Sep 05 03:34:39 retryDelay: 20000 Sep 05 03:34:39 } Sep 05 03:34:39 Sep 05 03:34:39 Node.js v20.18.0 Sep 05 03:34:41 box:shell backuptask: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_[UUID] tgz {"localRoot":"/home/yellowtent/appsdata/[UUID]","layout":[]} errored BoxError: backuptask exited with code 1 signal null Sep 05 03:34:41 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19) Sep 05 03:34:41 at ChildProcess.emit (node:events:519:28) Sep 05 03:34:41 at ChildProcess.emit (node:domain:488:12) Sep 05 03:34:41 at ChildProcess._handle.onexit (node:internal/child_process:294:12) { Sep 05 03:34:41 reason: 'Shell Error', Sep 05 03:34:41 details: {}, Sep 05 03:34:41 code: 1, Sep 05 03:34:41 signal: null Sep 05 03:34:41 } Sep 05 03:34:41 box:backuptask runBackupUpload: backuptask crashed BoxError: backuptask exited with code 1 signal null Sep 05 03:34:41 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19) Sep 05 03:34:41 at ChildProcess.emit (node:events:519:28) Sep 05 03:34:41 at ChildProcess.emit (node:domain:488:12) Sep 05 03:34:41 at ChildProcess._handle.onexit (node:internal/child_process:294:12) { Sep 05 03:34:41 reason: 'Shell Error', Sep 05 03:34:41 details: {}, Sep 05 03:34:41 code: 1, Sep 05 03:34:41 signal: null Sep 05 03:34:41 } Sep 05 03:34:41 box:backuptask fullBackup: app [REDACTED] backup finished. Took 331.629 seconds Sep 05 03:34:41 box:locks write: current locks: {"backup_task":null} Sep 05 03:34:41 box:locks release: app_[UUID] Sep 05 03:34:41 box:taskworker Task took 2080.827 seconds Sep 05 03:34:41 box:tasks setCompleted - 7013: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}} Sep 05 03:34:41 box:tasks update 7013: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}} Sep 05 03:34:41 BoxError: Backuptask crashed Sep 05 03:34:41 at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15) Sep 05 03:34:41 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) Sep 05 03:34:41 at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5) Sep 05 03:34:41 at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)