error while backing up

[Jarod]

Any idea? Maybe a bug?

Dec 05 23:12:02 box:backuptask rotateMailBackup: rotating to 2024-12-05-220000-794/mail_v8.1.0
Dec 05 23:12:02 box:tasks update 2014: {"percent":90.47368421052636,"message":"Copying with concurrency of 10"}
Dec 05 23:12:04 box:tasks update 2014: {"percent":90.47368421052636,"message":"Copying files from 0-1"}
Dec 05 23:12:04 box:tasks update 2014: {"percent":90.47368421052636,"message":"Copying (multipart) snapshot/mail.tar.gz"}
Dec 05 23:12:04 box:tasks update 2014: {"percent":90.47368421052636,"message":"Copying part 1 - /cloudronbackup-REMOVEDFORFORUM/snapshot/mail.tar.gz bytes=0-1073741823"}
Dec 05 23:12:04 box:tasks update 2014: {"percent":90.47368421052636,"message":"Copying part 2 - /cloudronbackup-REMOVEDFORFORUM/snapshot/mail.tar.gz bytes=1073741824-2147483647"}
Dec 05 23:12:04 box:tasks update 2014: {"percent":90.47368421052636,"message":"Copying part 3 - /cloudronbackup-REMOVEDFORFORUM/snapshot/mail.tar.gz bytes=2147483648-3221225471"}
Dec 05 23:13:04 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (1) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:13:04 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (1) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:13:04 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (1) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:14:23 box:tasks update 2014: {"percent":90.47368421052636,"message":"Copying part 2 - Etag: 77e882104ff405cdbf8d091053169ca7"}
Dec 05 23:14:23 box:tasks update 2014: {"percent":90.47368421052636,"message":"Copying part 4 - /cloudronbackup-REMOVEDFORFORUM/snapshot/mail.tar.gz bytes=3221225472-4294967295"}
Dec 05 23:14:25 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (2) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:14:25 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (2) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:15:23 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (1) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:15:45 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (3) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:15:45 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (3) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:16:43 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (2) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:17:05 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (4) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:17:05 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (4) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:18:03 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (3) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:18:25 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (5) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:18:25 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (5) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:19:23 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (4) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:19:45 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (6) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:19:45 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (6) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:20:43 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (5) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:21:05 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (7) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:21:05 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (7) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:22:03 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (6) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:22:25 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (8) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:22:25 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (8) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:23:23 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (7) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:23:45 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (9) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:23:45 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (9) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:24:41 box:tasks update 2014: {"percent":90.47368421052636,"message":"Copying part 4 - Etag: a2132197a4d42261637fc197ed066b13"}
Dec 05 23:24:41 box:tasks update 2014: {"percent":90.47368421052636,"message":"Copying part 5 - /cloudronbackup-REMOVEDFORFORUM/snapshot/mail.tar.gz bytes=4294967296-4776642292"}
Dec 05 23:25:05 box:tasks update 2014: {"percent":90.47368421052636,"message":"Copying part 1 - Etag: f42130621cb6ce886373fc94e97a0940"}
Dec 05 23:25:05 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (10) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:25:09 box:tasks update 2014: {"percent":90.47368421052636,"message":"Copying part 5 - Etag: e60e0408e3be489be7986b759a8f3606"}
Dec 05 23:26:25 box:tasks update 2014: {"percent":90.47368421052636,"message":"Retrying (11) multipart copy of snapshot/mail.tar.gz. Error: 504: null 504"}
Dec 05 23:26:25 box:tasks update 2014: {"percent":90.47368421052636,"message":"Aborting multipart copy of snapshot/mail.tar.gz"}
Dec 05 23:26:28 box:storage/s3 copy: s3 copy error when copying snapshot/mail.tar.gz: 504: null
Dec 05 23:26:28 box:tasks update 2014: {"percent":90.47368421052636,"message":"Copied 1 files with error: BoxError: Error copying snapshot/mail.tar.gz (4776642293 bytes): 504 504: null"}
Dec 05 23:26:28 box:backuptask copy: copied to 2024-12-05-220000-794/mail_v8.1.0 errored. error: Error copying snapshot/mail.tar.gz (4776642293 bytes): 504 504: null
Dec 05 23:26:28 box:taskworker Task took 1587.812 seconds
Dec 05 23:26:28 box:tasks setCompleted - 2014: {"result":null,"error":{"stack":"BoxError: Error copying snapshot/mail.tar.gz (4776642293 bytes): 504 504: null\n at done (/home/yellowtent/box/src/storage/s3.js:333:48)\n at Response.<anonymous> (/home/yellowtent/box/src/storage/s3.js:409:71)\n at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:367:18)\n at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)\n at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:686:14)\n at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)\n at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)\n at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10\n at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9)","name":"BoxError","reason":"External Error","details":{},"message":"Error copying snapshot/mail.tar.gz (4776642293 bytes): 504 504: null"}}
Dec 05 23:26:28 box:tasks update 2014: {"percent":100,"result":null,"error":{"stack":"BoxError: Error copying snapshot/mail.tar.gz (4776642293 bytes): 504 504: null\n at done (/home/yellowtent/box/src/storage/s3.js:333:48)\n at Response.<anonymous> (/home/yellowtent/box/src/storage/s3.js:409:71)\n at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:367:18)\n at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)\n at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:686:14)\n at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)\n at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)\n at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10\n at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9)","name":"BoxError","reason":"External Error","details":{},"message":"Error copying snapshot/mail.tar.gz (4776642293 bytes): 504 504: null"}}
Dec 05 23:26:28 BoxError: Error copying snapshot/mail.tar.gz (4776642293 bytes): 504 504: null
Dec 05 23:26:28 at done (/home/yellowtent/box/src/storage/s3.js:333:48)
Dec 05 23:26:28 at Response.<anonymous> (/home/yellowtent/box/src/storage/s3.js:409:71)
Dec 05 23:26:28 llowtent/box/node_modules/aws-sdk/lib/request.js:367:18)
Dec 05 23:26:28 at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
Dec 05 23:26:28 at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
Dec 05 23:26:28 at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:686:14)
Dec 05 23:26:28 at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)
Dec 05 23:26:28 at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)
Dec 05 23:26:28 at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10
Dec 05 23:26:28 at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9)

[nebulon]

Is this happening all the time? The error status code returned from Hetzner is 504 indicating an issue on the objects storage side. Cloudron here retries a few times but eventually gives up.

[girish]

Which region of Hetzner is this? I just tested on HEL1 and multi-part copies is fine.

[Dave Swift]

Also happening for me -- Falkenstein (FSN1).

[girish]

@Dave-Swift this happens all the time? If it's sporadic, then I suspect this is just a Hetzner issue (are they still in beta?). Curiously, I am not finding any announcement related to the stability/availability .

[Dave Swift]

It was happening across multiple servers at the same time yesterday. Looks to be resolved now.