Error 400 in backup process with Ionos S3 Object Storage

[p44]

@dsp76 did you try to debug other S3 storage provider?

[dsp76]

I don't have others...

[dsp76]

The logging doesn't show, that its related to the external system. More like an error in the internal backup task in cloudron?

[dsp76]

Its still happening. Everyday a different app in backup process. At different times.
@nebulon could we please remove the "solved" tag? I'm happy to help with debugging, if I know what to look for.
I'm now also looking into timing and potentially overlaps with other tasks.

[dsp76]

Moved it to another time, made sure no update processes are overlapping. Still:

"Backuptask crashed\n at runBackupUpload"

As mentioned before - there was no error in the log at Ionos S3.

The root cause may be this:

Sep 10 04:00:28 box:shell backuptask: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_UUID tgz {"localRoot":"/home/yellowtent/appsdata/UUID","layout":[]} errored BoxError: backuptask exited with code 1 signal null
Sep 10 04:00:28 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19)
Sep 10 04:00:28 at ChildProcess.emit (node:events:519:28)
Sep 10 04:00:28 at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
Sep 10 04:00:28 reason: 'Shell Error',
Sep 10 04:00:28 details: {},
Sep 10 04:00:28 code: 1,
Sep 10 04:00:28 signal: null

System log:

{
  "taskId": "7060",
  "errorMessage": "Backuptask crashed",
  "timedOut": false,
  "backupId": null
}

The Problem: The system Backup log still shows the complete number of apps (supposed to be backed up). Checking them shows, they havn't been backed up. Looks like, that instead of only one task crashed, the complete backup task crashed and ended the complete operation in the same minute it started, @nebulon ...

[joseph]

@dsp76 I have marked this as unsolved. Is there nothing else in the task/backup logs? i.e right above the crash messages.

[dsp76]

Hi @joseph - the complete log is in #8 https://forum.cloudron.io/topic/14253/error-400-in-backup-process-with-ionos-s3-object-storage/8?_=1757661856982

This is todays error log of the backup task:

Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg3.jpg file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-1024x335.jpg file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-1200x393.jpg file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-150x150.jpg file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-1536x502.jpg file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-177x142.jpg file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-200x65.jpg file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-300x214.jpg file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-300x98.jpg file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-320x202.jpg file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-400x131.jpg file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-460x295.jpg file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-540x272.jpg file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-600x196.jpg file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-669x272.jpg file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-66x66.jpg file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-700x441.jpg file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-768x251.jpg file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-800x262.jpg file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-940x400.jpg file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4.jpg file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-1024x342.png file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-1200x401.png file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-150x150.png file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-1536x513.png file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-177x142.png file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-200x67.png file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-2048x684.png file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-300x100.png file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-300x214.png file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-320x202.png file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-400x134.png file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-460x295.png file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-540x272.png file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-600x200.png file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-669x272.png file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-66x66.png file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-700x441.png file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-768x256.png file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-800x267.png file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-940x400.png file
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-scaled.jpg file
Sep 12 04:17:21 /home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:712
Sep 12 04:17:21 resp.error = AWS.util.error(new Error(), {
Sep 12 04:17:21 ^
Sep 12 04:17:21
Sep 12 04:17:21 400: null
Sep 12 04:17:21 at Request.extractError (/home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:712:35)
Sep 12 04:17:21 at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
Sep 12 04:17:21 at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
Sep 12 04:17:21 at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:686:14)
Sep 12 04:17:21 at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)
Sep 12 04:17:21 at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)
Sep 12 04:17:21 at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10
Sep 12 04:17:21 at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9)
Sep 12 04:17:21 at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:688:12)
Sep 12 04:17:21 at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
Sep 12 04:17:21 code: 400,
Sep 12 04:17:21 region: null,
Sep 12 04:17:21 time: 2025-09-12T02:17:21.579Z,
Sep 12 04:17:21 requestId: null,
Sep 12 04:17:21 extendedRequestId: undefined,
Sep 12 04:17:21 cfId: undefined,
Sep 12 04:17:21 statusCode: 400,
Sep 12 04:17:21 retryable: false,
Sep 12 04:17:21 retryDelay: 20000
Sep 12 04:17:21 }
Sep 12 04:17:21
Sep 12 04:17:21 Node.js v20.18.0
Sep 12 04:17:21 box:shell backuptask: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_{UUID} tgz {"localRoot":"/home/yellowtent/appsdata/{UUID}","layout":[]} errored BoxError: backuptask exited with code 1 signal null
Sep 12 04:17:21 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19)
Sep 12 04:17:21 at ChildProcess.emit (node:events:519:28)
Sep 12 04:17:21 at ChildProcess.emit (node:domain:488:12)
Sep 12 04:17:21 at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
Sep 12 04:17:21 reason: 'Shell Error',
Sep 12 04:17:21 details: {},
Sep 12 04:17:21 code: 1,
Sep 12 04:17:21 signal: null
Sep 12 04:17:21 }
Sep 12 04:17:21 box:backuptask runBackupUpload: backuptask crashed BoxError: backuptask exited with code 1 signal null
Sep 12 04:17:21 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19)
Sep 12 04:17:21 at ChildProcess.emit (node:events:519:28)
Sep 12 04:17:21 at ChildProcess.emit (node:domain:488:12)
Sep 12 04:17:21 at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
Sep 12 04:17:21 reason: 'Shell Error',
Sep 12 04:17:21 details: {},
Sep 12 04:17:21 code: 1,
Sep 12 04:17:21 signal: null
Sep 12 04:17:21 }
Sep 12 04:17:21 box:backuptask fullBackup: app {DOMAIN} backup finished. Took 364.294 seconds
Sep 12 04:17:21 box:locks write: current locks: {"backup_task":null}
Sep 12 04:17:21 box:locks release: app_{UUID}
Sep 12 04:17:21 box:taskworker Task took 1041.04 seconds
Sep 12 04:17:21 box:tasks setCompleted - 7083: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
Sep 12 04:17:21 box:tasks update 7083: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
Sep 12 04:17:21 BoxError: Backuptask crashed
Sep 12 04:17:21 at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)
Sep 12 04:17:21 at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Sep 12 04:17:21 at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)
Sep 12 04:17:21 at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)

[dsp76]

It also crashes when I start it manually.

[dsp76]

Switching back to SSHFS in Hetzner Storage Box seems to run the backup reliable.
So whats wrong with S3 in combination with Ionos then? It looks like the process dies in cloudron, but the error log for the task itself seems not to help. Ionos found no issues in the log.

[james]

Hello @dsp76
Good to read that you found a solution for you that works for now.
Regarding the issues with IONOS S3, we will have to look into it.

Just read the following topic Backup to Hetzner Object Storage failing regularly and now reading that you have no issues with Hetzner SSHFS.
It seems the experience between users are worlds apart.

Maybe some insights about:

• amount of apps
• what apps
• tgz or rsync
• backup size
• schedule
• full encryption
• file name encryption

from you @dsp76, @p44 and @ccfu might reveal some common issues?