"Backup Failed" since the last Cloudron Update

[guyds]

I noticed similar issues since the update to Cloudron 8, but in my case it was caused by the fact that the backup storage wasn't mounted correctly.
After clicking the "remount storage" button this was fixed and backups started working again.

I'm using sshfs and had this issue on 2 Cloudrons.
Maybe there is a general regression in the update procedure that causes the backup storage location to be in an unmounted state, resulting in failed backups?

[joseph]

@guyds the sshfs issue is somehow related to DNS, but some users have reported this issue. Once you remount, the problem should be fixed thereafter.

@kubasbimbas not sure what's going on. Are you able to contact us on support@cloudron.io so we acn debug further?

[dellis]

I am also getting errors daily on backup. I use Backblaze B2. Here is a partial log:

2024-08-10T06:15:12.080Z box:storage/s3 Upload progress: {"loaded":2915041280,"part":274,"key":"snapshot/app_7481caee-b863-4501-8a36-4fb08884505c.tar.gz"}
2024-08-10T06:15:12.663Z box:storage/s3 Upload progress: {"loaded":2925527040,"part":281,"key":"snapshot/app_7481caee-b863-4501-8a36-4fb08884505c.tar.gz"}
2024-08-10T06:15:13.201Z box:storage/s3 Upload progress: {"loaded":2936012800,"part":282,"key":"snapshot/app_7481caee-b863-4501-8a36-4fb08884505c.tar.gz"}
2024-08-10T06:15:13.835Z box:storage/s3 Upload progress: {"loaded":2946498560,"part":283,"key":"snapshot/app_7481caee-b863-4501-8a36-4fb08884505c.tar.gz"}
2024-08-10T06:15:14.397Z box:storage/s3 Upload progress: {"loaded":2956984320,"part":284,"key":"snapshot/app_7481caee-b863-4501-8a36-4fb08884505c.tar.gz"}
2024-08-10T06:15:15.154Z box:storage/s3 Upload progress: {"loaded":2967470080,"part":285,"key":"snapshot/app_7481caee-b863-4501-8a36-4fb08884505c.tar.gz"}
2024-08-10T06:15:16.550Z box:tasks update 2092: {"percent":34.333333333333336,"message":"Uploading backup 2876M@7MBps (nc.dellis.co)"}
2024-08-10T06:15:21.116Z box:storage/s3 Upload progress: {"loaded":2967470080,"part":274,"key":"snapshot/app_7481caee-b863-4501-8a36-4fb08884505c.tar.gz"}
node:events:496
throw er; // Unhandled 'error' event
^

Error: write EPIPE
at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)
Emitted 'error' event on TLSSocket instance at:
at emitErrorNT (node:internal/streams/destroy:169:8)
at emitErrorCloseNT (node:internal/streams/destroy:128:3)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
errno: -32,
code: 'EPIPE',
syscall: 'write'
}

Node.js v20.12.2
2024-08-10T06:15:21.150Z box:shell backup-snapshot/app_7481caee-b863-4501-8a36-4fb08884505c: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_7481caee-b863-4501-8a36-4fb08884505c tgz {"localRoot":"/home/yellowtent/appsdata/7481caee-b863-4501-8a36-4fb08884505c","layout":[]} errored BoxError: backup-snapshot/app_7481caee-b863-4501-8a36-4fb08884505c exited with code 1 signal null
at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:122:19)
at ChildProcess.emit (node:events:518:28)
at ChildProcess.emit (node:domain:488:12)
at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
reason: 'Shell Error',
details: {},
code: 1,
signal: null
}
2024-08-10T06:15:21.154Z box:taskworker Task took 919.866 seconds
2024-08-10T06:15:21.164Z box:tasks setCompleted - 2092: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
2024-08-10T06:15:21.165Z box:tasks update 2092: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
BoxError: Backuptask crashed
at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)
at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)
at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)

[Wenhardt]

I've been running into the same error as @dellis recently and I also use Backblaze B2. I even tried switching to rsync and that didn't fix the issue. I haven't been able to back up at all.

This started before v8.02.

[Dont-Worry]

Again today. And i changed the S3 from AWS to Backblaze. Manual backup is working but automatic doesn't work. But since the change have been done, it is like automatic update doesn't trigger all the time.. (The successful backup has been triggered by me manualy).

Full logs of the last automatic failed backup (from today).

Aug 14 04:04:12 box:backupformat/tgz addToPack: added ./data/data/admin/files/EM/M1-S2/Rapport d'Activité/Rapport d'Activité n°2.docx file
Aug 14 04:04:12 box:backupformat/tgz tarPack: processing /mnt/volumes/6e6d77d1ac41447da93a95677f20c922/Drive/data/admin/files/IA
Aug 14 04:04:12 box:backupformat/tgz addToPack: added ./data/data/admin/files/IA/Ressources directory
Aug 14 04:04:12 box:backupformat/tgz tarPack: processing /mnt/volumes/6e6d77d1ac41447da93a95677f20c922/Drive/data/admin/files/IA/Ressources
Aug 14 04:04:12 box:backupformat/tgz addToPack: added ./data/data/admin/files/IA/Ressources/SOP pour Vérifier l'Activité d'une Application sur GitHub.docx file
Aug 14 04:04:12 box:backupformat/tgz tarPack: processing /mnt/volumes/6e6d77d1ac41447da93a95677f20c922/Drive/data/admin/files/Perso
Aug 14 04:04:12 box:backupformat/tgz addToPack: added ./data/data/admin/files/Perso/Articles A Acheter directory
Aug 14 04:04:12 box:backupformat/tgz addToPack: added ./data/data/admin/files/Perso/Cadeaux Idées directory
Aug 14 04:04:12 box:backupformat/tgz addToPack: added ./data/data/admin/files/Perso/Casquettes directory
Aug 14 04:04:12 box:backupformat/tgz addToPack: added ./data/data/admin/files/Perso/Dress 15-06-24 directory
Aug 14 04:04:12 box:backupformat/tgz addToPack: added ./data/data/admin/files/Perso/Elle.docx file
Aug 14 04:04:12 box:backupformat/tgz addToPack: added ./data/data/admin/files/Perso/Jeux Vidéos directory
Aug 14 04:04:12 box:backupformat/tgz addToPack: added ./data/data/admin/files/Perso/Logo Propal 2 A-One.png file
Aug 14 04:04:13 box:storage/s3 Upload progress: {"loaded":3221225472,"part":2,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"}
Aug 14 04:04:13 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 4539M@44MBps (drive.example.com)"}
Aug 14 04:04:14 box:backupformat/tgz addToPack: added ./data/data/admin/files/Perso/Monologue Rayane Insta.mp4 file
Aug 14 04:04:14 box:backupformat/tgz addToPack: added ./data/data/admin/files/Perso/Musiques directory
Aug 14 04:04:23 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 5009M@47MBps (drive.example.com)"}
Aug 14 04:04:33 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 5400M@39MBps (drive.example.com)"}
Aug 14 04:04:43 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 5860M@46MBps (drive.example.com)"}
Aug 14 04:04:53 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 6144M@28MBps (drive.example.com)"}
Aug 14 04:04:53 box:storage/s3 Upload progress: {"loaded":4294967296,"part":5,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"}
Aug 14 04:05:03 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 6144M@0MBps (drive.example.com)"}
Aug 14 04:05:13 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 6144M@0MBps (drive.example.com)"}
Aug 14 04:05:19 box:storage/s3 Upload progress: {"loaded":4294967296,"part":5,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"}
Aug 14 04:05:23 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 6144M@0MBps (drive.example.com)"}
Aug 14 04:05:33 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 6144M@0MBps (drive.example.com)"}
Aug 14 04:05:34 box:storage/s3 Upload progress: {"loaded":4294967296,"part":1,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"}
Aug 14 04:05:34 box:storage/s3 Upload progress: {"loaded":5368709120,"part":4,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"}
Aug 14 04:05:44 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 6144M@0MBps (drive.example.com)"}
Aug 14 04:05:52 box:storage/s3 Upload progress: {"loaded":5368709120,"part":5,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"}
Aug 14 04:05:54 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 6583M@44MBps (drive.example.com)"}
Aug 14 04:06:04 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 7038M@46MBps (drive.example.com)"}
Aug 14 04:06:14 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 7420M@38MBps (drive.example.com)"}
Aug 14 04:06:23 box:storage/s3 Upload progress: {"loaded":5368709120,"part":5,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"}
Aug 14 04:06:24 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 7891M@47MBps (drive.example.com)"}
Aug 14 04:06:34 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 8192M@30MBps (drive.example.com)"}
Aug 14 04:06:44 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 8192M@0MBps (drive.example.com)"}
Aug 14 04:06:54 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 8192M@0MBps (drive.example.com)"}
Aug 14 04:07:04 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 8192M@0MBps (drive.example.com)"}
Aug 14 04:07:14 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 8192M@0MBps (drive.example.com)"}
Aug 14 04:07:16 box:storage/s3 Upload progress: {"loaded":6442450944,"part":6,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"}
Aug 14 04:07:24 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 8435M@24MBps (drive.example.com)"}
Aug 14 04:07:34 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 8894M@46MBps (drive.example.com)"}
Aug 14 04:07:43 box:storage/s3 Upload progress: {"loaded":7516192768,"part":8,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"}
Aug 14 04:07:44 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 9216M@32MBps (drive.example.com)"}
Aug 14 04:07:54 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 9216M@0MBps (drive.example.com)"}
Aug 14 04:08:01 box:storage/s3 Upload progress: {"loaded":7516192768,"part":5,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"}
Aug 14 04:08:04 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 9217M@0MBps (drive.example.com)"}
Aug 14 04:08:14 box:storage/s3 Upload progress: {"loaded":7516192768,"part":8,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"}
Aug 14 04:08:14 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 9685M@47MBps (drive.example.com)"}
Aug 14 04:08:24 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 10143M@46MBps (drive.example.com)"}
Aug 14 04:08:34 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 10240M@10MBps (drive.example.com)"}
Aug 14 04:08:44 box:storage/s3 Upload progress: {"loaded":7516192768,"part":8,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"}
Aug 14 04:08:44 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 10240M@0MBps (drive.example.com)"}
Aug 14 04:08:54 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 10240M@0MBps (drive.example.com)"}
Aug 14 04:09:04 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 10240M@0MBps (drive.example.com)"}
Aug 14 04:09:05 box:storage/s3 Upload progress: {"loaded":8589934592,"part":7,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"}
Aug 14 04:09:14 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 10240M@0MBps (drive.example.com)"}
Aug 14 04:09:24 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 10554M@31MBps (drive.example.com)"}
Aug 14 04:09:32 box:storage/s3 Upload progress: {"loaded":9663676416,"part":9,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"}
Aug 14 04:09:34 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 11023M@47MBps (drive.example.com)"}
Aug 14 04:09:44 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 11416M@39MBps (drive.example.com)"}
Aug 14 04:09:54 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 11864M@45MBps (drive.example.com)"}
Aug 14 04:09:59 box:storage/s3 Upload progress: {"loaded":10737418240,"part":10,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"}
Jan 01 00:00:00 node:events:496
[no timestamp]  throw er; // Unhandled 'error' event
[no timestamp]  ^
[no timestamp] 
[no timestamp]  write EPIPE
[no timestamp]  at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)
[no timestamp]  'error' event on TLSSocket instance at:
[no timestamp]  at emitErrorNT (node:internal/streams/destroy:169:8)
[no timestamp]  at emitErrorCloseNT (node:internal/streams/destroy:128:3)
[no timestamp]  at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
[no timestamp]  errno: -32,
[no timestamp]  code: 'EPIPE',
[no timestamp]  syscall: 'write'
[no timestamp]  }
[no timestamp] 
[no timestamp]  v20.12.2
Aug 14 04:09:59 box:shell backup-snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872 tgz {"localRoot":"/home/yellowtent/appsdata/370be07e-c281-4d35-95a1-5c9984f72872","layout":[{"localDir":"/mnt/volumes/6e6d77d1ac41447da93a95677f20c922/Drive","remoteDir":"data"}]} errored BoxError: backup-snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872 exited with code 1 signal null
[no timestamp]  at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:122:19)
[no timestamp]  at ChildProcess.emit (node:events:518:28)
[no timestamp]  at ChildProcess.emit (node:domain:488:12)
[no timestamp]  at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
[no timestamp]  reason: 'Shell Error',
[no timestamp]  details: {},
[no timestamp]  code: 1,
[no timestamp]  signal: null
[no timestamp]  }
Aug 14 04:09:59 box:taskworker Task took 598.935 seconds
Aug 14 04:09:59 box:tasks setCompleted - 3103: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
Aug 14 04:09:59 box:tasks update 3103: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
[no timestamp]  Backuptask crashed
[no timestamp]  at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15)
[no timestamp]  at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[no timestamp]  at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)
[no timestamp]  at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)
[no timestamp]  at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)

[kubasbimbas]

I have noticed that the backup issue affects also the app updates, as the app is backed up before it is updated. Last night the Immich app started scheduled update 2 times from v1.68.1, but never updated to v1.69.0. I have then started the update manually this morning and it was completed.

[joseph]

@kubasbimbas that is expected, apps are always backed up before an update. if backups fails, the update will abort and go back to previous version .

[kubasbimbas]

@joseph I have verified that the app data were uploaded to the backup backend. Let's wait for tonight scheduled update to v1.69.1.

[girish]

Just looked into this. I think there is a regression in Cloudron 8. We fixed the backup cleaner to work properly but in the process it cleans up the backup entry in the database that is being created. https://git.cloudron.io/cloudron/box/-/commit/9704eefc21af313a4239e74999186ae44a43ed46 is the fix for that. @kubasbimbas can you try that one line change? You have to add that line to the file /home/yellowtent/box/src/backupcleaner.js on the server . No need to restart anything after applying the patch.

[Dont-Worry]

@girish Hi,

What do I have to do to fix it ? I don't understand what I have to do with the git link you sent.

Thanks

[girish]

@Dont-Worry no worries. if you send us a mail to support@cloudron.io , we can apply the patch on your server . (We debugged the issue on somebody's server, but we cannot match forum username and support requests... If you had sent us a mail ealier, we already patched your server)

[kubasbimbas]

@girish You patched my server. I confirm that automatic Cloudron backup as well as app update worked correctly. Thank you for your support.

[potemkin_ai]

Same here on multiple servers backing up against S3. What's an ETA for the hotfix release?

[girish]

@potemkin_ai we will make a 8.0.4 next week .

[potemkin_ai]

If that's not too much, I would appreciate if you could ping this thread, once the new version released.

[girish]

@potemkin_ai is out now

[potemkin_ai]

Thank you!

[potemkin_ai]

@girish , Cloudron v8.0.4, backups are still failing:

Aug 28 03:00:04 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/<...>
[no timestamp]  /home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:712
[no timestamp]  resp.error = AWS.util.error(new Error(), {
[no timestamp]  ^
[no timestamp] 
[no timestamp]  The specified method is not allowed against this resource.
[no timestamp]  at Request.extractError (/home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:712:35)
[no timestamp]  at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
[no timestamp]  at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
[no timestamp]  at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:686:14)
[no timestamp]  at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)
[no timestamp]  at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)
[no timestamp]  at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10
[no timestamp]  at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9)
[no timestamp]  at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:688:12)
[no timestamp]  at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
[no timestamp]  code: 'MethodNotAllowed',
[no timestamp]  region: null,
[no timestamp]  time: 2024-08-28T01:00:04.170Z,
[no timestamp]  requestId: '45dc6280-1336-4ef2-82df-8bf7545a1d89',
[no timestamp]  extendedRequestId: '45dc6280-1336-4ef2-82df-8bf7545a1d89',
[no timestamp]  cfId: undefined,
[no timestamp]  statusCode: 405,
[no timestamp]  retryable: false,
[no timestamp]  retryDelay: 20000
[no timestamp]  }
[no timestamp] 
[no timestamp]  v20.12.2

The provider is Exoscale's SOS.

[potemkin_ai]

@girish , I'm sorry, is there any news on that? I still can't backup my instance.

[nebulon]

It seems to crash within the AWS SDK itself, I guess your S3 storage has some compatibility issues there, we have seen such problem in the past where the storage endpoint was flaky.

Maybe you can contact exoscale about this? MethodNotAllowed might indicate that they don't implement/support some required API?