Error 400 in backup process with Ionos S3 Object Storage
-
Its still happening. Everyday a different app in backup process. At different times.
@nebulon could we please remove the "solved" tag? I'm happy to help with debugging, if I know what to look for.
I'm now also looking into timing and potentially overlaps with other tasks. -
Moved it to another time, made sure no update processes are overlapping. Still:
"Backuptask crashed\n at runBackupUpload"
As mentioned before - there was no error in the log at Ionos S3.
The root cause may be this:
Sep 10 04:00:28 box:shell backuptask: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_UUID tgz {"localRoot":"/home/yellowtent/appsdata/UUID","layout":[]} errored BoxError: backuptask exited with code 1 signal null Sep 10 04:00:28 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19) Sep 10 04:00:28 at ChildProcess.emit (node:events:519:28) Sep 10 04:00:28 at ChildProcess._handle.onexit (node:internal/child_process:294:12) { Sep 10 04:00:28 reason: 'Shell Error', Sep 10 04:00:28 details: {}, Sep 10 04:00:28 code: 1, Sep 10 04:00:28 signal: nullSystem log:
{ "taskId": "7060", "errorMessage": "Backuptask crashed", "timedOut": false, "backupId": null }The Problem: The system Backup log still shows the complete number of apps (supposed to be backed up). Checking them shows, they havn't been backed up. Looks like, that instead of only one task crashed, the complete backup task crashed and ended the complete operation in the same minute it started, @nebulon ...
-
J joseph has marked this topic as unsolved on
-
Hi @joseph - the complete log is in #8 https://forum.cloudron.io/topic/14253/error-400-in-backup-process-with-ionos-s3-object-storage/8?_=1757661856982
This is todays error log of the backup task:
Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg3.jpg file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-1024x335.jpg file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-1200x393.jpg file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-150x150.jpg file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-1536x502.jpg file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-177x142.jpg file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-200x65.jpg file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-300x214.jpg file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-300x98.jpg file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-320x202.jpg file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-400x131.jpg file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-460x295.jpg file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-540x272.jpg file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-600x196.jpg file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-669x272.jpg file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-66x66.jpg file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-700x441.jpg file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-768x251.jpg file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-800x262.jpg file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4-940x400.jpg file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg4.jpg file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-1024x342.png file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-1200x401.png file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-150x150.png file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-1536x513.png file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-177x142.png file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-200x67.png file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-2048x684.png file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-300x100.png file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-300x214.png file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-320x202.png file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-400x134.png file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-460x295.png file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-540x272.png file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-600x200.png file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-669x272.png file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-66x66.png file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-700x441.png file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-768x256.png file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-800x267.png file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-940x400.png file Sep 12 04:17:21 box:backupformat/tgz addToPack: added ./data/public/wp-content/uploads/2024/03/testbg5-scaled.jpg file Sep 12 04:17:21 /home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:712 Sep 12 04:17:21 resp.error = AWS.util.error(new Error(), { Sep 12 04:17:21 ^ Sep 12 04:17:21 Sep 12 04:17:21 400: null Sep 12 04:17:21 at Request.extractError (/home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:712:35) Sep 12 04:17:21 at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20) Sep 12 04:17:21 at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10) Sep 12 04:17:21 at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:686:14) Sep 12 04:17:21 at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10) Sep 12 04:17:21 at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12) Sep 12 04:17:21 at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10 Sep 12 04:17:21 at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9) Sep 12 04:17:21 at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:688:12) Sep 12 04:17:21 at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:116:18) { Sep 12 04:17:21 code: 400, Sep 12 04:17:21 region: null, Sep 12 04:17:21 time: 2025-09-12T02:17:21.579Z, Sep 12 04:17:21 requestId: null, Sep 12 04:17:21 extendedRequestId: undefined, Sep 12 04:17:21 cfId: undefined, Sep 12 04:17:21 statusCode: 400, Sep 12 04:17:21 retryable: false, Sep 12 04:17:21 retryDelay: 20000 Sep 12 04:17:21 } Sep 12 04:17:21 Sep 12 04:17:21 Node.js v20.18.0 Sep 12 04:17:21 box:shell backuptask: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_{UUID} tgz {"localRoot":"/home/yellowtent/appsdata/{UUID}","layout":[]} errored BoxError: backuptask exited with code 1 signal null Sep 12 04:17:21 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19) Sep 12 04:17:21 at ChildProcess.emit (node:events:519:28) Sep 12 04:17:21 at ChildProcess.emit (node:domain:488:12) Sep 12 04:17:21 at ChildProcess._handle.onexit (node:internal/child_process:294:12) { Sep 12 04:17:21 reason: 'Shell Error', Sep 12 04:17:21 details: {}, Sep 12 04:17:21 code: 1, Sep 12 04:17:21 signal: null Sep 12 04:17:21 } Sep 12 04:17:21 box:backuptask runBackupUpload: backuptask crashed BoxError: backuptask exited with code 1 signal null Sep 12 04:17:21 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19) Sep 12 04:17:21 at ChildProcess.emit (node:events:519:28) Sep 12 04:17:21 at ChildProcess.emit (node:domain:488:12) Sep 12 04:17:21 at ChildProcess._handle.onexit (node:internal/child_process:294:12) { Sep 12 04:17:21 reason: 'Shell Error', Sep 12 04:17:21 details: {}, Sep 12 04:17:21 code: 1, Sep 12 04:17:21 signal: null Sep 12 04:17:21 } Sep 12 04:17:21 box:backuptask fullBackup: app {DOMAIN} backup finished. Took 364.294 seconds Sep 12 04:17:21 box:locks write: current locks: {"backup_task":null} Sep 12 04:17:21 box:locks release: app_{UUID} Sep 12 04:17:21 box:taskworker Task took 1041.04 seconds Sep 12 04:17:21 box:tasks setCompleted - 7083: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}} Sep 12 04:17:21 box:tasks update 7083: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}} Sep 12 04:17:21 BoxError: Backuptask crashed Sep 12 04:17:21 at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15) Sep 12 04:17:21 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) Sep 12 04:17:21 at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5) Sep 12 04:17:21 at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5) -
Switching back to SSHFS in Hetzner Storage Box seems to run the backup reliable.
So whats wrong with S3 in combination with Ionos then? It looks like the process dies in cloudron, but the error log for the task itself seems not to help. Ionos found no issues in the log. -
Hello @dsp76
Good to read that you found a solution for you that works for now.
Regarding the issues with IONOS S3, we will have to look into it.Just read the following topic Backup to Hetzner Object Storage failing regularly and now reading that you have no issues with Hetzner SSHFS.
It seems the experience between users are worlds apart.Maybe some insights about:
- amount of apps
- what apps
- tgz or rsync
- backup size
- schedule
- full encryption
- file name encryption
from you @dsp76, @p44 and @ccfu might reveal some common issues?
-
Hello @dsp76
Good to read that you found a solution for you that works for now.
Regarding the issues with IONOS S3, we will have to look into it.Just read the following topic Backup to Hetzner Object Storage failing regularly and now reading that you have no issues with Hetzner SSHFS.
It seems the experience between users are worlds apart.Maybe some insights about:
- amount of apps
- what apps
- tgz or rsync
- backup size
- schedule
- full encryption
- file name encryption
from you @dsp76, @p44 and @ccfu might reveal some common issues?
In this case, it seems that something is interfering with backup scripts... what I would like to test – if possible – to stop all the apps, and retry with a "lighter" backup to see what will happens.
I've suspect that problem is from Hetzner side, but, of course, need more search to prove this.
Because of lacking of time, when I experienced issues (eg. unable to restore backup, different errors), I moved to other Isp and solved all the problems.
-
Hello @p44
If we can narrow it down like this, we have contacts to the Hetzner Team that is working on the Storage Boxes and could relay these details for further investigation.@james Yes, I totally agree with your point of view.
But as a paying customer and with a final user perspective I would not rely on that service, due to my past experience...
Of course, this is only my experience with Hetzner and this don’t means that all users are experiencing the same issues...
-
I had the same 400 error with IONOS S3 and it turned out to be a simple setup issue. In Cloudron’s backup settings, make sure you select the IONOS (Profitbricks) option instead of the generic S3 choice. Then double-check that the endpoint and region match your bucket exactly—for example s3.<your-region>.ionoscloud.com—and that path-style URLs are enabled. After I corrected those settings, the backups ran without any errors. If you still get the 400 after that, try a small manual backup or upload a tiny file with the same keys using the AWS CLI to confirm the credentials. If the problem persists, sharing the few log lines just before the crash will help others spot what’s wrong.
Hope this helps get your backups running smoothly.
-
Hello @dsp76
Good to read that you found a solution for you that works for now.
Regarding the issues with IONOS S3, we will have to look into it.Just read the following topic Backup to Hetzner Object Storage failing regularly and now reading that you have no issues with Hetzner SSHFS.
It seems the experience between users are worlds apart.Maybe some insights about:
- amount of apps
- what apps
- tgz or rsync
- backup size
- schedule
- full encryption
- file name encryption
from you @dsp76, @p44 and @ccfu might reveal some common issues?
Hi @james ,
Hetzner Object Storage ≠Hetzner Storage Box (thats what I was using as alternative backup target). I didn't try the Object Storage.@halima632 that was a good hint! Just switched to the IONOS (Profitbricks) entry and it seems to run fine. I didn't know it existed and thought it was "S3 compatible" and fine
-
D dsp76 has marked this topic as solved on
-
Unfortunately even with the correct IONOS Settings from my Hetzner based Cloudron.
It started crashing again every night since a week now. Thats the anonymised log:
Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/index.html file Oct 15 04:12:36 box:backupformat/tgz tarPack: processing /home/***/appsdata/********/data/public/wp-content/cache/borlabs-cookie Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/borlabs-cookie/1 directory Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/borlabs-cookie/borlabs-cookie_1_de.css file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/borlabs-cookie/borlabs-cookie_1_en.css file Oct 15 04:12:37 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/borlabs-cookie/geo-ip-database-********.mmdb file Oct 15 04:12:37 box:backupformat/tgz tarPack: processing /home/***/appsdata/********/data/public/wp-content/cache/borlabs-cookie/1 Oct 15 04:12:37 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/borlabs-cookie/1/borlabs-cookie-1-de.css file Oct 15 04:12:37 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/borlabs-cookie/1/borlabs-cookie-1-en.css file Oct 15 04:12:37 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/borlabs-cookie/1/borlabs-cookie-config-de.json.js file Oct 15 04:12:37 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/borlabs-cookie/1/borlabs-cookie-config-en.json.js file Oct 15 04:12:37 box:backupformat/tgz tarPack: processing /home/***/appsdata/********/data/public/wp-content/cache/matomo Oct 15 04:12:37 box:backupformat/tgz tarPack: processing /home/***/appsdata/********/data/public/wp-content/infinitewp Oct 15 04:12:37 box:backupformat/tgz addToPack: added ./data/public/wp-content/infinitewp/backups directory Oct 15 04:12:37 box:backupformat/tgz addToPack: added ./data/public/wp-content/infinitewp/temp directory Oct 15 04:12:37 box:backupformat/tgz tarPack: processing /home/***/appsdata/********/data/public/wp-content/infinitewp/backups Oct 15 04:12:37 box:backupformat/tgz addToPack: added ./data/public/wp-content/infinitewp/backups/DE_clCPUUsage.****.txt file Oct 15 04:12:37 box:backupformat/tgz addToPack: added ./data/public/wp-content/infinitewp/backups/DE_clMemoryPeak.****.txt file Oct 15 04:12:37 box:backupformat/tgz addToPack: added ./data/public/wp-content/infinitewp/backups/DE_clMemoryUsage.****.txt file Oct 15 04:12:37 box:backupformat/tgz addToPack: added ./data/public/wp-content/infinitewp/backups/DE_clTimeTaken.****.txt file Oct 15 04:12:37 box:backupformat/tgz addToPack: added ./data/public/wp-content/infinitewp/backups/index.php file Oct 15 04:12:42 box:tasks update ****: {"percent":21.3,"message":"Uploading backup 134M@9MBps (example.com)"} Oct 15 04:12:42 box:storage/s3 Upload progress: {"loaded":134217728,"part":1,"key":"snapshot/app_********.tar.gz.enc"} Oct 15 04:12:51 box:storage/s3 Upload progress: {"loaded":268435456,"part":2,"key":"snapshot/app_********.tar.gz.enc"} Oct 15 04:12:52 box:tasks update ****: {"percent":21.3,"message":"Uploading backup 277M@14MBps (example.com)"} Oct 15 04:13:02 box:tasks update ****: {"percent":21.3,"message":"Uploading backup 435M@16MBps (example.com)"} Oct 15 04:13:03 box:backupformat/tgz addToPack: added ./data/public/wp-content/infinitewp/backups/example.com-de_backup_full_****-**-**_********.zip file Oct 15 04:13:08 box:storage/s3 Upload progress: {"loaded":402653184,"part":4,"key":"snapshot/app_********.tar.gz.enc"} Oct 15 04:13:12 box:tasks update ****: {"percent":21.3,"message":"Uploading backup 584M@15MBps (example.com)"} Oct 15 04:13:12 AWS SDK Error: 400 Bad Request Oct 15 04:13:12 code: 400, statusCode: 400, retryable: false Oct 15 04:13:12 Node.js v20.18.0 Oct 15 04:13:19 box:shell backuptask: errored BoxError: backuptask exited with code 1 signal null Oct 15 04:13:19 reason: 'Shell Error', code: 1 Oct 15 04:13:19 box:backuptask runBackupUpload: backuptask crashed BoxError: backuptask exited with code 1 signal null Oct 15 04:13:19 box:backuptask fullBackup: app example.com backup finished. Took 67.438 seconds Oct 15 04:13:19 box:tasks update ****: {"percent":100,"error":{"message":"Backuptask crashed"}} Oct 15 04:13:19 BoxError: Backuptask crashedNot sure what to do with it. Just started a manual backup, it also crashed again while uploading to IONOS.
Oct 15 11:00:05 400: null Oct 15 11:00:05 at Request.extractError (/home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:712:35) Oct 15 11:00:05 at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20) Oct 15 11:00:05 at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10) Oct 15 11:00:05 at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:686:14) Oct 15 11:00:05 at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10) Oct 15 11:00:05 at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12) Oct 15 11:00:05 at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10 Oct 15 11:00:05 at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9) Oct 15 11:00:05 at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:688:12) Oct 15 11:00:05 at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:116:18) { Oct 15 11:00:05 code: 400, Oct 15 11:00:05 region: null, Oct 15 11:00:05 time: 2025-10-15T09:00:05.031Z, Oct 15 11:00:05 requestId: null, Oct 15 11:00:05 extendedRequestId: undefined, Oct 15 11:00:05 cfId: undefined, Oct 15 11:00:05 statusCode: 400, Oct 15 11:00:05 retryable: false, Oct 15 11:00:05 retryDelay: 20000 Oct 15 11:00:05 }and
Oct 15 11:00:05 box:tasks setCompleted - 7460: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}} Oct 15 11:00:05 box:tasks update 7460: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}} Oct 15 11:00:05 BoxError: Backuptask crashedThe snapshot folder within the bucket (created by cloudron?) contains a couple of up to date uploads.
- Who should I with - is it IONOS? Is it cloudron? Is it Hetzner?
- Whats causing the "internal error" while uploading?
- Why is it "retryable: false"?
-
J james has marked this topic as unsolved
-
Unfortunately even with the correct IONOS Settings from my Hetzner based Cloudron.
It started crashing again every night since a week now. Thats the anonymised log:
Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/autoptimize_snippet_********.js file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/autoptimize/js/index.html file Oct 15 04:12:36 box:backupformat/tgz tarPack: processing /home/***/appsdata/********/data/public/wp-content/cache/borlabs-cookie Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/borlabs-cookie/1 directory Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/borlabs-cookie/borlabs-cookie_1_de.css file Oct 15 04:12:36 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/borlabs-cookie/borlabs-cookie_1_en.css file Oct 15 04:12:37 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/borlabs-cookie/geo-ip-database-********.mmdb file Oct 15 04:12:37 box:backupformat/tgz tarPack: processing /home/***/appsdata/********/data/public/wp-content/cache/borlabs-cookie/1 Oct 15 04:12:37 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/borlabs-cookie/1/borlabs-cookie-1-de.css file Oct 15 04:12:37 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/borlabs-cookie/1/borlabs-cookie-1-en.css file Oct 15 04:12:37 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/borlabs-cookie/1/borlabs-cookie-config-de.json.js file Oct 15 04:12:37 box:backupformat/tgz addToPack: added ./data/public/wp-content/cache/borlabs-cookie/1/borlabs-cookie-config-en.json.js file Oct 15 04:12:37 box:backupformat/tgz tarPack: processing /home/***/appsdata/********/data/public/wp-content/cache/matomo Oct 15 04:12:37 box:backupformat/tgz tarPack: processing /home/***/appsdata/********/data/public/wp-content/infinitewp Oct 15 04:12:37 box:backupformat/tgz addToPack: added ./data/public/wp-content/infinitewp/backups directory Oct 15 04:12:37 box:backupformat/tgz addToPack: added ./data/public/wp-content/infinitewp/temp directory Oct 15 04:12:37 box:backupformat/tgz tarPack: processing /home/***/appsdata/********/data/public/wp-content/infinitewp/backups Oct 15 04:12:37 box:backupformat/tgz addToPack: added ./data/public/wp-content/infinitewp/backups/DE_clCPUUsage.****.txt file Oct 15 04:12:37 box:backupformat/tgz addToPack: added ./data/public/wp-content/infinitewp/backups/DE_clMemoryPeak.****.txt file Oct 15 04:12:37 box:backupformat/tgz addToPack: added ./data/public/wp-content/infinitewp/backups/DE_clMemoryUsage.****.txt file Oct 15 04:12:37 box:backupformat/tgz addToPack: added ./data/public/wp-content/infinitewp/backups/DE_clTimeTaken.****.txt file Oct 15 04:12:37 box:backupformat/tgz addToPack: added ./data/public/wp-content/infinitewp/backups/index.php file Oct 15 04:12:42 box:tasks update ****: {"percent":21.3,"message":"Uploading backup 134M@9MBps (example.com)"} Oct 15 04:12:42 box:storage/s3 Upload progress: {"loaded":134217728,"part":1,"key":"snapshot/app_********.tar.gz.enc"} Oct 15 04:12:51 box:storage/s3 Upload progress: {"loaded":268435456,"part":2,"key":"snapshot/app_********.tar.gz.enc"} Oct 15 04:12:52 box:tasks update ****: {"percent":21.3,"message":"Uploading backup 277M@14MBps (example.com)"} Oct 15 04:13:02 box:tasks update ****: {"percent":21.3,"message":"Uploading backup 435M@16MBps (example.com)"} Oct 15 04:13:03 box:backupformat/tgz addToPack: added ./data/public/wp-content/infinitewp/backups/example.com-de_backup_full_****-**-**_********.zip file Oct 15 04:13:08 box:storage/s3 Upload progress: {"loaded":402653184,"part":4,"key":"snapshot/app_********.tar.gz.enc"} Oct 15 04:13:12 box:tasks update ****: {"percent":21.3,"message":"Uploading backup 584M@15MBps (example.com)"} Oct 15 04:13:12 AWS SDK Error: 400 Bad Request Oct 15 04:13:12 code: 400, statusCode: 400, retryable: false Oct 15 04:13:12 Node.js v20.18.0 Oct 15 04:13:19 box:shell backuptask: errored BoxError: backuptask exited with code 1 signal null Oct 15 04:13:19 reason: 'Shell Error', code: 1 Oct 15 04:13:19 box:backuptask runBackupUpload: backuptask crashed BoxError: backuptask exited with code 1 signal null Oct 15 04:13:19 box:backuptask fullBackup: app example.com backup finished. Took 67.438 seconds Oct 15 04:13:19 box:tasks update ****: {"percent":100,"error":{"message":"Backuptask crashed"}} Oct 15 04:13:19 BoxError: Backuptask crashedNot sure what to do with it. Just started a manual backup, it also crashed again while uploading to IONOS.
Oct 15 11:00:05 400: null Oct 15 11:00:05 at Request.extractError (/home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:712:35) Oct 15 11:00:05 at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20) Oct 15 11:00:05 at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10) Oct 15 11:00:05 at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:686:14) Oct 15 11:00:05 at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10) Oct 15 11:00:05 at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12) Oct 15 11:00:05 at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10 Oct 15 11:00:05 at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9) Oct 15 11:00:05 at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:688:12) Oct 15 11:00:05 at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:116:18) { Oct 15 11:00:05 code: 400, Oct 15 11:00:05 region: null, Oct 15 11:00:05 time: 2025-10-15T09:00:05.031Z, Oct 15 11:00:05 requestId: null, Oct 15 11:00:05 extendedRequestId: undefined, Oct 15 11:00:05 cfId: undefined, Oct 15 11:00:05 statusCode: 400, Oct 15 11:00:05 retryable: false, Oct 15 11:00:05 retryDelay: 20000 Oct 15 11:00:05 }and
Oct 15 11:00:05 box:tasks setCompleted - 7460: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}} Oct 15 11:00:05 box:tasks update 7460: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}} Oct 15 11:00:05 BoxError: Backuptask crashedThe snapshot folder within the bucket (created by cloudron?) contains a couple of up to date uploads.
- Who should I with - is it IONOS? Is it cloudron? Is it Hetzner?
- Whats causing the "internal error" while uploading?
- Why is it "retryable: false"?
@dsp76 the retryable comes the aws sdk . It's indicating whether we should retry a failing request or not. Since retryable is false, the code gives up immediately.
Can you write to us at support@cloudron.io ? We have to debug this further to understand which request is failing and why .
-
Hello @dsp76 I have marked to topic as unsolved again.
Where you unable to do this yourself or was this more of a courtesy question? -
I've had the same problem since a week ago. The strange thing is that everything had been running flawlessly for a year. However, we recently set up a second Cloudron. The backup is stored on a different bucket and also has its own key. The problem occurs on both servers. Unfortunately, it's not deterministic.
03:00:33 box:storage/s3 Upload progress: {"loaded":95400493056,"part":920,"key":"backup/snapshot/app_da39dd94-29b5-4049-9aa5-76864ebc4608.tar.gz.enc"} Oct 16 03:00:33 /home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:712 Oct 16 03:00:33 resp.error = AWS.util.error(new Error(), { Oct 16 03:00:33 ^ Oct 16 03:00:33 Oct 16 03:00:33 400: null Oct 16 03:00:33 at Request.extractError (/home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:712:35) Oct 16 03:00:33 at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20) Oct 16 03:00:33 at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10) Oct 16 03:00:33 at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:686:14) Oct 16 03:00:33 at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10) Oct 16 03:00:33 at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12) Oct 16 03:00:33 at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10 Oct 16 03:00:33 at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9) Oct 16 03:00:33 at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:688:12) Oct 16 03:00:33 at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:116:18) { Oct 16 03:00:33 code: 400, Oct 16 03:00:33 region: null, Oct 16 03:00:33 time: 2025-10-16T01:00:34.559Z, Oct 16 03:00:33 requestId: null, Oct 16 03:00:33 extendedRequestId: undefined, Oct 16 03:00:33 cfId: undefined, Oct 16 03:00:33 statusCode: 400, Oct 16 03:00:33 retryable: false, Oct 16 03:00:33 retryDelay: 20000 Oct 16 03:00:33 } Oct 16 03:00:33 Oct 16 03:00:33 Node.js v20.18.0 Oct 16 03:00:34 box:shell backuptask: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_da39dd94-29b5-4049-9aa5-76864ebc4608 tgz {"localRoot":"/home/yellowtent/appsdata/da39dd94-29b5-4049-9aa5-76864ebc4608","layout":[]} errored BoxError: backuptask exited with code 1 signal null Oct 16 03:00:34 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19) Oct 16 03:00:34 at ChildProcess.emit (node:events:519:28) Oct 16 03:00:34 at ChildProcess.emit (node:domain:488:12) Oct 16 03:00:34 at ChildProcess._handle.onexit (node:internal/child_process:294:12) { Oct 16 03:00:34 reason: 'Shell Error', Oct 16 03:00:34 details: {}, Oct 16 03:00:34 code: 1, Oct 16 03:00:34 signal: null Oct 16 03:00:34 } Oct 16 03:00:34 box:backuptask runBackupUpload: backuptask crashed BoxError: backuptask exited with code 1 signal null Oct 16 03:00:34 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19) Oct 16 03:00:34 at ChildProcess.emit (node:events:519:28) Oct 16 03:00:34 at ChildProcess.emit (node:domain:488:12) Oct 16 03:00:34 at ChildProcess._handle.onexit (node:internal/child_process:294:12) { Oct 16 03:00:34 reason: 'Shell Error', Oct 16 03:00:34 details: {}, Oct 16 03:00:34 code: 1, Oct 16 03:00:34 signal: null Oct 16 03:00:34 } -
I've had the same problem since a week ago. The strange thing is that everything had been running flawlessly for a year. However, we recently set up a second Cloudron. The backup is stored on a different bucket and also has its own key. The problem occurs on both servers. Unfortunately, it's not deterministic.
03:00:33 box:storage/s3 Upload progress: {"loaded":95400493056,"part":920,"key":"backup/snapshot/app_da39dd94-29b5-4049-9aa5-76864ebc4608.tar.gz.enc"} Oct 16 03:00:33 /home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:712 Oct 16 03:00:33 resp.error = AWS.util.error(new Error(), { Oct 16 03:00:33 ^ Oct 16 03:00:33 Oct 16 03:00:33 400: null Oct 16 03:00:33 at Request.extractError (/home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:712:35) Oct 16 03:00:33 at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20) Oct 16 03:00:33 at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10) Oct 16 03:00:33 at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:686:14) Oct 16 03:00:33 at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10) Oct 16 03:00:33 at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12) Oct 16 03:00:33 at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10 Oct 16 03:00:33 at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9) Oct 16 03:00:33 at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:688:12) Oct 16 03:00:33 at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:116:18) { Oct 16 03:00:33 code: 400, Oct 16 03:00:33 region: null, Oct 16 03:00:33 time: 2025-10-16T01:00:34.559Z, Oct 16 03:00:33 requestId: null, Oct 16 03:00:33 extendedRequestId: undefined, Oct 16 03:00:33 cfId: undefined, Oct 16 03:00:33 statusCode: 400, Oct 16 03:00:33 retryable: false, Oct 16 03:00:33 retryDelay: 20000 Oct 16 03:00:33 } Oct 16 03:00:33 Oct 16 03:00:33 Node.js v20.18.0 Oct 16 03:00:34 box:shell backuptask: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_da39dd94-29b5-4049-9aa5-76864ebc4608 tgz {"localRoot":"/home/yellowtent/appsdata/da39dd94-29b5-4049-9aa5-76864ebc4608","layout":[]} errored BoxError: backuptask exited with code 1 signal null Oct 16 03:00:34 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19) Oct 16 03:00:34 at ChildProcess.emit (node:events:519:28) Oct 16 03:00:34 at ChildProcess.emit (node:domain:488:12) Oct 16 03:00:34 at ChildProcess._handle.onexit (node:internal/child_process:294:12) { Oct 16 03:00:34 reason: 'Shell Error', Oct 16 03:00:34 details: {}, Oct 16 03:00:34 code: 1, Oct 16 03:00:34 signal: null Oct 16 03:00:34 } Oct 16 03:00:34 box:backuptask runBackupUpload: backuptask crashed BoxError: backuptask exited with code 1 signal null Oct 16 03:00:34 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19) Oct 16 03:00:34 at ChildProcess.emit (node:events:519:28) Oct 16 03:00:34 at ChildProcess.emit (node:domain:488:12) Oct 16 03:00:34 at ChildProcess._handle.onexit (node:internal/child_process:294:12) { Oct 16 03:00:34 reason: 'Shell Error', Oct 16 03:00:34 details: {}, Oct 16 03:00:34 code: 1, Oct 16 03:00:34 signal: null Oct 16 03:00:34 }
