Backup problem, not clear what is it

[chetbaker]

Hello,

I'm having this odd error time to time with my backups. I'm using backblazeb2 for that and as you can see in the screenshot sometimes it works, sometimes it doesn't.

I don't know if others are suffering something similar to this to guide me because the logs are obscure to me:

ct 21 06:45:32 box:tasks update 12024: {"percent":21.408163265306126,"message":"Retrying (11) multipart copy of snapshot/app_3ff951ec-5941-43d1-8918-79c262a9d436.tar.gz.enc. Error: InternalError: incident id d4937d31ad81-05ae1c2c1f984baa 500"}
Oct 21 06:45:32 box:tasks update 12024: {"percent":21.408163265306126,"message":"Aborting multipart copy of snapshot/app_3ff951ec-5941-43d1-8918-79c262a9d436.tar.gz.enc"}
Oct 21 06:45:33 box:storage/s3 copy: s3 copy error when copying snapshot/app_3ff951ec-5941-43d1-8918-79c262a9d436.tar.gz.enc: InternalError: incident id d4937d31ad81-05ae1c2c1f984baa
Oct 21 06:45:33 box:tasks update 12024: {"percent":21.408163265306126,"message":"Copied 1 files with error: BoxError: Error copying snapshot/app_3ff951ec-5941-43d1-8918-79c262a9d436.tar.gz.enc (15934368724 bytes): InternalError InternalError: incident id d4937d31ad81-05ae1c2c1f984baa"}
Oct 21 06:45:33 box:backuptask copy: copied to 2024-10-21-070000-627/app_cloud.criti.ca_v4.22.5 errored. error: Error copying snapshot/app_3ff951ec-5941-43d1-8918-79c262a9d436.tar.gz.enc (15934368724 bytes): InternalError InternalError: incident id d4937d31ad81-05ae1c2c1f984baa
Oct 21 06:45:33 box:taskworker Task took 9932.641 seconds
Oct 21 06:45:33 box:tasks setCompleted - 12024: {"result":null,"error":{"stack":"BoxError: Error copying snapshot/app_3ff951ec-5941-43d1-8918-79c262a9d436.tar.gz.enc (15934368724 bytes): InternalError InternalError: incident id d4937d31ad81-05ae1c2c1f984baa\n at done (/home/yellowtent/box/src/storage/s3.js:333:48)\n at Response.<anonymous> (/home/yellowtent/box/src/storage/s3.js:409:71)\n at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:367:18)\n at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)\n at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:686:14)\n at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)\n at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)\n at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10\n at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9)","name":"BoxError","reason":"External Error","details":{},"message":"Error copying snapshot/app_3ff951ec-5941-43d1-8918-79c262a9d436.tar.gz.enc (15934368724 bytes): InternalError InternalError: incident id d4937d31ad81-05ae1c2c1f984baa"}}
Oct 21 06:45:33 box:tasks update 12024: {"percent":100,"result":null,"error":{"stack":"BoxError: Error copying snapshot/app_3ff951ec-5941-43d1-8918-79c262a9d436.tar.gz.enc (15934368724 bytes): InternalError InternalError: incident id d4937d31ad81-05ae1c2c1f984baa\n at done (/home/yellowtent/box/src/storage/s3.js:333:48)\n at Response.<anonymous> (/home/yellowtent/box/src/storage/s3.js:409:71)\n at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:367:18)\n at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)\n at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:686:14)\n at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)\n at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)\n at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10\n at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9)","name":"BoxError","reason":"External Error","details":{},"message":"Error copying snapshot/app_3ff951ec-5941-43d1-8918-79c262a9d436.tar.gz.enc (15934368724 bytes): InternalError InternalError: incident id d4937d31ad81-05ae1c2c1f984baa"}}
[no timestamp]  Error copying snapshot/app_3ff951ec-5941-43d1-8918-79c262a9d436.tar.gz.enc (15934368724 bytes): InternalError InternalError: incident id d4937d31ad81-05ae1c2c1f984baa
[no timestamp]  at done (/home/yellowtent/box/src/storage/s3.js:333:48)
[no timestamp]  at Response.<anonymous> (/home/yellowtent/box/src/storage/s3.js:409:71)
[no timestamp]  at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:367:18)
[no timestamp]  at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
[no timestamp]  at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
[no timestamp]  at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:686:14)
[no timestamp]  at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)
[no timestamp]  at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)
[no timestamp]  at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10
[no timestamp]  at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9)

[nebulon]

It seems backblaze is having a tough time at the moment. We have encountered various temporary issues with it lately. Also this one seems like backblaze is returning some Internal Error. So as long as it eventually works out for the moment, lets see if they recover.

[chetbaker]

I'm not sure whether this is related, but if it is, it will be out of interest of all who uses Backlaze for backups:

[nebulon]

Ah this is good information and indeed a likely a problem for rsync backups with a lot of files (not sure how the rate-limit turns out in the end though)