Error copying backup/snapshot/app_....tar.gz Access Denied Access Denied: null

[taylorstevens]

Hi folks. My backups started failing for me a few weeks ago and I hadn't made any changes to my setup in quite some time, other than automatic updates.

When manually trying to create a backup, I get the following error:

Error copying backup/snapshot/app_751f52fb-328b-45a3-aeb4-d735fabbc0cd.tar.gz (5110809205 bytes): AccessDenied AccessDenied: null

I saw in another thread that an older version of Ubuntu could be the cause, so I migrated to 22.04, but I'm having the same problem.

My backups are configured for Object Storage on Linode. I rotated my access keys just to make sure something wasn't amiss there. I think specifically, the Nextcloud backup fails, I'm seeing other apps successfully backup.

I'm wondering if things are timing out? Anyone see this before? Here's the full log:

Mar 29 23:02:27box:tasks update 5252: {"percent":38.5,"message":"Uploading backup 1486M@17MBps (cloud.andystevens.name)"}

...

Mar 29 23:07:07box:tasks update 5252: {"percent":38.5,"message":"Uploading backup 4850M@9MBps (cloud.andystevens.name)"}
Mar 29 23:07:11box:backupupload upload completed. error: null
Mar 29 23:07:11box:backuptask runBackupUpload: result - {"result":""}
Mar 29 23:07:11box:shell backup-snapshot/app_751f52fb-328b-45a3-aeb4-d735fabbc0cd (stderr): 2024-03-30T03:07:11.310Z box:storage/s3 Uploaded backup/snapshot/app_751f52fb-328b-45a3-aeb4-d735fabbc0cd.tar.gz with partSize 10485760: {"Location":"us-east-1.linodeobjects.com/my.andystevens.name/backup/snapshot/app_751f52fb-328b-45a3-aeb4-d735fabbc0cd.tar.gz","Bucket":"my.andystevens.name","Key":"backup/snapshot/app_751f52fb-328b-45a3-aeb4-d735fabbc0cd.tar.gz","ETag":"e97baf9601fafa4388146bd545bcadf7-488"}
Mar 29 23:07:11box:backuptask uploadAppSnapshot: cloud.andystevens.name uploaded to snapshot/app_751f52fb-328b-45a3-aeb4-d735fabbc0cd. 425.666 seconds
Mar 29 23:07:11box:backuptask rotateAppBackup: rotating cloud.andystevens.name to path 2024-03-30-030001-032/app_cloud.andystevens.name_v4.21.1
Mar 29 23:07:11box:tasks update 5252: {"percent":38.5,"message":"Copying with concurrency of 10"}
Mar 29 23:07:11box:tasks update 5252: {"percent":38.5,"message":"Copying files from 0-1"}
Mar 29 23:07:11box:tasks update 5252: {"percent":38.5,"message":"Copying (multipart) backup/snapshot/app_751f52fb-328b-45a3-aeb4-d735fabbc0cd.tar.gz"}
Mar 29 23:07:11box:tasks update 5252: {"percent":38.5,"message":"Copying part 1 - /my.andystevens.name/backup/snapshot/app_751f52fb-328b-45a3-aeb4-d735fabbc0cd.tar.gz bytes=0-1073741823"}
Mar 29 23:07:11box:tasks update 5252: {"percent":38.5,"message":"Copying part 2 - /my.andystevens.name/backup/snapshot/app_751f52fb-328b-45a3-aeb4-d735fabbc0cd.tar.gz bytes=1073741824-2147483647"}
Mar 29 23:07:11box:tasks update 5252: {"percent":38.5,"message":"Copying part 3 - /my.andystevens.name/backup/snapshot/app_751f52fb-328b-45a3-aeb4-d735fabbc0cd.tar.gz bytes=2147483648-3221225471"}
Mar 29 23:07:11box:tasks update 5252: {"percent":38.5,"message":"Retrying (1) multipart copy of backup/snapshot/app_751f52fb-328b-45a3-aeb4-d735fabbc0cd.tar.gz. Error: AccessDenied: null 403"}
Mar 29 23:07:11box:tasks update 5252: {"percent":38.5,"message":"Aborting multipart copy of backup/snapshot/app_751f52fb-328b-45a3-aeb4-d735fabbc0cd.tar.gz"}
Mar 29 23:07:11box:storage/s3 copy: s3 copy error when copying backup/snapshot/app_751f52fb-328b-45a3-aeb4-d735fabbc0cd.tar.gz: AccessDenied: null
Mar 29 23:07:11box:backuptask copy: copied to 2024-03-30-030001-032/app_cloud.andystevens.name_v4.21.1 errored. error: Error copying backup/snapshot/app_751f52fb-328b-45a3-aeb4-d735fabbc0cd.tar.gz (5110807012 bytes): AccessDenied AccessDenied: null
Mar 29 23:07:11box:tasks update 5252: {"percent":38.5,"message":"Copied 1 files with error: BoxError: Error copying backup/snapshot/app_751f52fb-328b-45a3-aeb4-d735fabbc0cd.tar.gz (5110807012 bytes): AccessDenied AccessDenied: null"}
Mar 29 23:07:11box:taskworker Task took 430.53 seconds
Mar 29 23:07:11box:tasks setCompleted - 5252: {"result":null,"error":{"stack":"BoxError: Error copying backup/snapshot/app_751f52fb-328b-45a3-aeb4-d735fabbc0cd.tar.gz (5110807012 bytes): AccessDenied AccessDenied: null\n at done (/home/yellowtent/box/src/storage/s3.js:338:48)\n at Response.<anonymous> (/home/yellowtent/box/src/storage/s3.js:414:71)\n at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:367:18)\n at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)\n at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:686:14)\n at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)\n at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)\n at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10\n at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9)","name":"BoxError","reason":"External Error","details":{},"message":"Error copying backup/snapshot/app_751f52fb-328b-45a3-aeb4-d735fabbc0cd.tar.gz (5110807012 bytes): AccessDenied AccessDenied: null"}}
Mar 29 23:07:11box:tasks update 5252: {"percent":100,"result":null,"error":{"stack":"BoxError: Error copying backup/snapshot/app_751f52fb-328b-45a3-aeb4-d735fabbc0cd.tar.gz (5110807012 bytes): AccessDenied AccessDenied: null\n at done (/home/yellowtent/box/src/storage/s3.js:338:48)\n at Response.<anonymous> (/home/yellowtent/box/src/storage/s3.js:414:71)\n at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:367:18)\n at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)\n at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:686:14)\n at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)\n at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)\n at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10\n at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9)","name":"BoxError","reason":"External Error","details":{},"message":"Error copying backup/snapshot/app_751f52fb-328b-45a3-aeb4-d735fabbc0cd.tar.gz (5110807012 bytes): AccessDenied AccessDenied: null"}}
Error copying backup/snapshot/app_751f52fb-328b-45a3-aeb4-d735fabbc0cd.tar.gz (5110807012 bytes): AccessDenied AccessDenied: null

[girish]

@taylorstevens it looks like multi-part file copy is failing on linode storage. Can you tell me which region you are using? I will test it myself here to see if I can reproduce it.

[taylorstevens]

@girish Thanks for taking a look. I'm using us-east-1 (Newark).

[girish]

@taylorstevens is this still a problem? Just got around to this and I was able to do a backup to a bucket in Newark (around 15gb of test data). I think the issue atleast per the error is something permission related.

[taylorstevens]

@girish yes it's still erroring. I'll try rotating keys again.

[taylorstevens]

@girish well, not sure why it didn't work before when I rotated my keys, but backups are back up.