"Backup Failed" since the last Cloudron Update
-
@girish , Cloudron v8.0.4, backups are still failing:
Aug 28 03:00:04 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/<...> [no timestamp] /home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:712 [no timestamp] resp.error = AWS.util.error(new Error(), { [no timestamp] ^ [no timestamp] [no timestamp] The specified method is not allowed against this resource. [no timestamp] at Request.extractError (/home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:712:35) [no timestamp] at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20) [no timestamp] at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10) [no timestamp] at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:686:14) [no timestamp] at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10) [no timestamp] at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12) [no timestamp] at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10 [no timestamp] at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9) [no timestamp] at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:688:12) [no timestamp] at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:116:18) { [no timestamp] code: 'MethodNotAllowed', [no timestamp] region: null, [no timestamp] time: 2024-08-28T01:00:04.170Z, [no timestamp] requestId: '45dc6280-1336-4ef2-82df-8bf7545a1d89', [no timestamp] extendedRequestId: '45dc6280-1336-4ef2-82df-8bf7545a1d89', [no timestamp] cfId: undefined, [no timestamp] statusCode: 405, [no timestamp] retryable: false, [no timestamp] retryDelay: 20000 [no timestamp] } [no timestamp] [no timestamp] v20.12.2The provider is Exoscale's SOS.
-
@girish , I'm sorry, is there any news on that? I still can't backup my instance.
-
It seems to crash within the AWS SDK itself, I guess your S3 storage has some compatibility issues there, we have seen such problem in the past where the storage endpoint was flaky.
Maybe you can contact exoscale about this?
MethodNotAllowedmight indicate that they don't implement/support some required API? -
@nebulon thank you! My other instance is running perfectly fine against the same endpoint and it was working fine on the same endpoint from the same instance before the update, so I'm not sure - how / why it could be an issue outside of Cloudron?
-
That I don't know either, but if I interpret the error correctly the AWS SDK gets a 405 error status contacting the storage backend. Which is the HTTP error for "Method Not Allowed" so that lines up at least. I would have no clue to even start debugging this from our side to be honest. Also I don't think this is related to this initial forum thread at all. Maybe you can create a new forum thread with as much info as possible to track this.
-
I have provider 'Exoscale' in both cases and the backup stopped working immediately after 8.0.3, and it doesn't yet.
It seems to be an issue on topic and I really can't do match - as everything is packed inside Exoscale module of Cloudron.
-
This AWS node module was updated at https://git.cloudron.io/cloudron/box/-/commit/b5fad74ea08e4c6cd06edf881612223549fb4eb3#fa288d1472d29beccb489a676f68739ad365fc47_14_14 so maybe that introduced the incompatibility with exoscale
So to be on the same page, your other Cloudron where backups are working, is on an older Cloudron version?
-
@nebulon thanks.
Both boxes are 8.0.4. Both back up to Exoscale.
The only difference I can see is that the one that works points to https://sos-ch-gva-2.exo.io, and the other (that doesn't) - to https://sos-at-vie-1.exo.io.I wouldn't expect the endpoints to be somehow different, as it would complicate operations for Exoscale, but that is really the only difference I can see.
-
G girish forked this topic on
-
@girish I tried single app backup and it produced a completely different error:
Sep 03 14:09:48 box:tasks update 4918: {"percent":30,"message":"Retrying (6) copy of ../../2024-08-06-010000-624/app_ps.domain_name_v1.17.0.tar.gz.enc. Error: IdenticalCopy: Source and copy are identical (contents and metadata). 400"} Sep 03 14:10:08 box:tasks update 4918: {"percent":30,"message":"Retrying (7) copy of ../../2024-08-06-010000-624/app_domain_name_v4.88.3.tar.gz.enc. Error: IdenticalCopy: Source and copy are identical (contents and metadata). 400"}It is also interesting to note, that I clicked backup on a single app and it's backing up all apps (that is just a snippet).
And - yeah - the error is different, so I guess we have a few of bunch of issues here.
-
I gave up with S3 - never liked a connected to it IAM, after all, tried to switch to Hetzner's storage box - it doesn't work as well, here are the logs:
Sep 04 12:24:30 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/5383ff24-feb3-47d1-b3e8-a0001f8832a8/data/templates/backup Sep 04 12:24:30 box:backupformat/tgz tarPack: pipeline finished: {"startTime":1725445433315,"totalSecs":37,"transferred":561899316} Sep 04 12:24:30 box:backupupload upload completed. error: null Sep 04 12:24:30 box:backuptask runBackupUpload: result - {"result":""} Sep 04 12:24:30 box:backuptask uploadAppSnapshot: team.domain.com uploaded to snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8. 37.813 seconds Sep 04 12:24:30 box:backuptask rotateAppBackup: rotating team.domain.com to path 2024-09-04-102351-773/app_team.domain.com_v1.1.1 Sep 04 12:24:30 box:tasks update 4939: {"percent":12.11111111111111,"message":"Copying /mnt/cloudronbackup/folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc to /mnt/cloudronbackup/folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc"} Sep 04 12:24:30 box:shell copy execArgs: ssh ["-o","\"StrictHostKeyChecking no\"","-i","/home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de","-p",22,"user@user.your-storagebox.de","cp","-al","folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc","folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc"] Sep 04 12:24:31 box:shell copy: ssh with args -o "StrictHostKeyChecking no" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc errored Error: Command failed: ssh -o "StrictHostKeyChecking no" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc [no timestamp] request failed on channel 0 [no timestamp] [no timestamp] at genericNodeError (node:internal/errors:984:15) [no timestamp] at wrappedFn (node:internal/errors:538:14) [no timestamp] at ChildProcess.exithandler (node:child_process:422:12) [no timestamp] at ChildProcess.emit (node:events:518:28) [no timestamp] at maybeClose (node:internal/child_process:1105:16) [no timestamp] at ChildProcess._handle.onexit (node:internal/child_process:305:5) { [no timestamp] code: 255, [no timestamp] killed: false, [no timestamp] signal: null, [no timestamp] cmd: 'ssh -o "StrictHostKeyChecking no" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc' [no timestamp] } Sep 04 12:24:31 box:backuptask copy: copied to 2024-09-04-102351-773/app_team.domain.com_v1.1.1 errored. error: SSH connection error: copy errored with code 255 message Command failed: ssh -o "StrictHostKeyChecking no" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc [no timestamp] request failed on channel 0 [no timestamp] Sep 04 12:24:31 box:taskworker Task took 39.381 seconds Sep 04 12:24:31 box:tasks setCompleted - 4939: {"result":null,"error":{"stack":"BoxError: SSH connection error: copy errored with code 255 message Command failed: ssh -o \"StrictHostKeyChecking no\" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc\nexec request failed on channel 0\r\n\n at Object.copy (/home/yellowtent/box/src/storage/filesystem.js:180:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)","name":"BoxError","reason":"External Error","details":{},"message":"SSH connection error: copy errored with code 255 message Command failed: ssh -o \"StrictHostKeyChecking no\" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc\nexec request failed on channel 0\r\n"}} Sep 04 12:24:31 box:tasks update 4939: {"percent":100,"result":null,"error":{"stack":"BoxError: SSH connection error: copy errored with code 255 message Command failed: ssh -o \"StrictHostKeyChecking no\" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc\nexec request failed on channel 0\r\n\n at Object.copy (/home/yellowtent/box/src/storage/filesystem.js:180:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)","name":"BoxError","reason":"External Error","details":{},"message":"SSH connection error: copy errored with code 255 message Command failed: ssh -o \"StrictHostKeyChecking no\" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc\nexec request failed on channel 0\r\n"}} [no timestamp] SSH connection error: copy errored with code 255 message Command failed: ssh -o "StrictHostKeyChecking no" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc [no timestamp] request failed on channel 0 [no timestamp] [no timestamp] at Object.copy (/home/yellowtent/box/src/storage/filesystem.js:180:49) [no timestamp] at process.processTicksAndRejections (node:internal/process/task_queues:95:5)The ssh command in question, when executed directly from the command line fails indeed with error message:
exec request failed on channel 0I've tried to allocate more memory to the backup process - just in case - but it didn't help.
Switching to rsync didn't produce any good results either - it's extremely slow and the logs are full with errors creating directory and files upload is just stalled.
SSHFS connection appears green, reconnection doesn't help. -
@potemkin_ai said in "Backup Failed" since the last Cloudron Update:
The ssh command in question, when executed directly from the command line fails indeed with error message: exec request failed on channel 0
Can you try port 23 of Hetzner? I haven't seen this error before though.
-
@girish , thanks - the port was the thing to blame here, thanks!
One more question, if you don't mind: I can see that sub-accounts are not recommended to use with SSHFS - is still actual with Cloudron v8, or it's a relic of the past?
