"Backup Failed" since the last Cloudron Update

[potemkin_ai]

Thank you!

[potemkin_ai]

@girish , Cloudron v8.0.4, backups are still failing:

Aug 28 03:00:04 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/<...>
[no timestamp]  /home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:712
[no timestamp]  resp.error = AWS.util.error(new Error(), {
[no timestamp]  ^
[no timestamp] 
[no timestamp]  The specified method is not allowed against this resource.
[no timestamp]  at Request.extractError (/home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:712:35)
[no timestamp]  at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
[no timestamp]  at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
[no timestamp]  at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:686:14)
[no timestamp]  at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)
[no timestamp]  at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)
[no timestamp]  at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10
[no timestamp]  at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9)
[no timestamp]  at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:688:12)
[no timestamp]  at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
[no timestamp]  code: 'MethodNotAllowed',
[no timestamp]  region: null,
[no timestamp]  time: 2024-08-28T01:00:04.170Z,
[no timestamp]  requestId: '45dc6280-1336-4ef2-82df-8bf7545a1d89',
[no timestamp]  extendedRequestId: '45dc6280-1336-4ef2-82df-8bf7545a1d89',
[no timestamp]  cfId: undefined,
[no timestamp]  statusCode: 405,
[no timestamp]  retryable: false,
[no timestamp]  retryDelay: 20000
[no timestamp]  }
[no timestamp] 
[no timestamp]  v20.12.2

The provider is Exoscale's SOS.

[potemkin_ai]

@girish , I'm sorry, is there any news on that? I still can't backup my instance.

[nebulon]

It seems to crash within the AWS SDK itself, I guess your S3 storage has some compatibility issues there, we have seen such problem in the past where the storage endpoint was flaky.

Maybe you can contact exoscale about this? MethodNotAllowed might indicate that they don't implement/support some required API?

[potemkin_ai]

@nebulon thank you! My other instance is running perfectly fine against the same endpoint and it was working fine on the same endpoint from the same instance before the update, so I'm not sure - how / why it could be an issue outside of Cloudron?

[nebulon]

That I don't know either, but if I interpret the error correctly the AWS SDK gets a 405 error status contacting the storage backend. Which is the HTTP error for "Method Not Allowed" so that lines up at least. I would have no clue to even start debugging this from our side to be honest. Also I don't think this is related to this initial forum thread at all. Maybe you can create a new forum thread with as much info as possible to track this.

[potemkin_ai]

I have provider 'Exoscale' in both cases and the backup stopped working immediately after 8.0.3, and it doesn't yet.

It seems to be an issue on topic and I really can't do match - as everything is packed inside Exoscale module of Cloudron.

[nebulon]

This AWS node module was updated at https://git.cloudron.io/cloudron/box/-/commit/b5fad74ea08e4c6cd06edf881612223549fb4eb3#fa288d1472d29beccb489a676f68739ad365fc47_14_14 so maybe that introduced the incompatibility with exoscale

So to be on the same page, your other Cloudron where backups are working, is on an older Cloudron version?

[girish]

I tested this with a test exoscale bucket in de-fra-1 region and it works. I wonder if this is IAM/Role related. I just gave it wholesale all permissions to objectstorage. Maybe you have something more specific?

[potemkin_ai]

@nebulon thanks.

Both boxes are 8.0.4. Both back up to Exoscale.
The only difference I can see is that the one that works points to https://sos-ch-gva-2.exo.io, and the other (that doesn't) - to https://sos-at-vie-1.exo.io.

I wouldn't expect the endpoints to be somehow different, as it would complicate operations for Exoscale, but that is really the only difference I can see.

[girish]

Test bucket in sos-at-vie-1.exo.io works as well . I am only testing with small backups if that matters. Does the small backup fail for you ? Like a single app? Maybe it's some multi-part stuff which is failing

[potemkin_ai]

@girish I tried single app backup and it produced a completely different error:

Sep 03 14:09:48 box:tasks update 4918: {"percent":30,"message":"Retrying (6) copy of ../../2024-08-06-010000-624/app_ps.domain_name_v1.17.0.tar.gz.enc. Error: IdenticalCopy: Source and copy are identical (contents and metadata). 400"}
Sep 03 14:10:08 box:tasks update 4918: {"percent":30,"message":"Retrying (7) copy of ../../2024-08-06-010000-624/app_domain_name_v4.88.3.tar.gz.enc. Error: IdenticalCopy: Source and copy are identical (contents and metadata). 400"}

It is also interesting to note, that I clicked backup on a single app and it's backing up all apps (that is just a snippet).

And - yeah - the error is different, so I guess we have a few of bunch of issues here.

[girish]

I see. I can't reproduce this on Exoscale SOS. Anyone else here use Exoscale?

Also, do you hit issues with other providers too or just Exoscale?

[potemkin_ai]

I gave up with S3 - never liked a connected to it IAM, after all, tried to switch to Hetzner's storage box - it doesn't work as well, here are the logs:

Sep 04 12:24:30 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/5383ff24-feb3-47d1-b3e8-a0001f8832a8/data/templates/backup
Sep 04 12:24:30 box:backupformat/tgz tarPack: pipeline finished: {"startTime":1725445433315,"totalSecs":37,"transferred":561899316}
Sep 04 12:24:30 box:backupupload upload completed. error: null
Sep 04 12:24:30 box:backuptask runBackupUpload: result - {"result":""}
Sep 04 12:24:30 box:backuptask uploadAppSnapshot: team.domain.com uploaded to snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8. 37.813 seconds
Sep 04 12:24:30 box:backuptask rotateAppBackup: rotating team.domain.com to path 2024-09-04-102351-773/app_team.domain.com_v1.1.1
Sep 04 12:24:30 box:tasks update 4939: {"percent":12.11111111111111,"message":"Copying /mnt/cloudronbackup/folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc to /mnt/cloudronbackup/folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc"}
Sep 04 12:24:30 box:shell copy execArgs: ssh ["-o","\"StrictHostKeyChecking no\"","-i","/home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de","-p",22,"user@user.your-storagebox.de","cp","-al","folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc","folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc"]
Sep 04 12:24:31 box:shell copy: ssh with args -o "StrictHostKeyChecking no" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc errored Error: Command failed: ssh -o "StrictHostKeyChecking no" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc
[no timestamp]  request failed on channel 0
[no timestamp] 
[no timestamp]  at genericNodeError (node:internal/errors:984:15)
[no timestamp]  at wrappedFn (node:internal/errors:538:14)
[no timestamp]  at ChildProcess.exithandler (node:child_process:422:12)
[no timestamp]  at ChildProcess.emit (node:events:518:28)
[no timestamp]  at maybeClose (node:internal/child_process:1105:16)
[no timestamp]  at ChildProcess._handle.onexit (node:internal/child_process:305:5) {
[no timestamp]  code: 255,
[no timestamp]  killed: false,
[no timestamp]  signal: null,
[no timestamp]  cmd: 'ssh -o "StrictHostKeyChecking no" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc'
[no timestamp]  }
Sep 04 12:24:31 box:backuptask copy: copied to 2024-09-04-102351-773/app_team.domain.com_v1.1.1 errored. error: SSH connection error: copy errored with code 255 message Command failed: ssh -o "StrictHostKeyChecking no" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc
[no timestamp]  request failed on channel 0
[no timestamp] 
Sep 04 12:24:31 box:taskworker Task took 39.381 seconds
Sep 04 12:24:31 box:tasks setCompleted - 4939: {"result":null,"error":{"stack":"BoxError: SSH connection error: copy errored with code 255 message Command failed: ssh -o \"StrictHostKeyChecking no\" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc\nexec request failed on channel 0\r\n\n at Object.copy (/home/yellowtent/box/src/storage/filesystem.js:180:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)","name":"BoxError","reason":"External Error","details":{},"message":"SSH connection error: copy errored with code 255 message Command failed: ssh -o \"StrictHostKeyChecking no\" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc\nexec request failed on channel 0\r\n"}}
Sep 04 12:24:31 box:tasks update 4939: {"percent":100,"result":null,"error":{"stack":"BoxError: SSH connection error: copy errored with code 255 message Command failed: ssh -o \"StrictHostKeyChecking no\" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc\nexec request failed on channel 0\r\n\n at Object.copy (/home/yellowtent/box/src/storage/filesystem.js:180:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)","name":"BoxError","reason":"External Error","details":{},"message":"SSH connection error: copy errored with code 255 message Command failed: ssh -o \"StrictHostKeyChecking no\" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc\nexec request failed on channel 0\r\n"}}
[no timestamp]  SSH connection error: copy errored with code 255 message Command failed: ssh -o "StrictHostKeyChecking no" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc
[no timestamp]  request failed on channel 0
[no timestamp] 
[no timestamp]  at Object.copy (/home/yellowtent/box/src/storage/filesystem.js:180:49)
[no timestamp]  at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

The ssh command in question, when executed directly from the command line fails indeed with error message: exec request failed on channel 0

I've tried to allocate more memory to the backup process - just in case - but it didn't help.

Switching to rsync didn't produce any good results either - it's extremely slow and the logs are full with errors creating directory and files upload is just stalled.
SSHFS connection appears green, reconnection doesn't help.

[girish]

@potemkin_ai said in "Backup Failed" since the last Cloudron Update:

The ssh command in question, when executed directly from the command line fails indeed with error message: exec request failed on channel 0

Can you try port 23 of Hetzner? I haven't seen this error before though.

[girish]

I can confirm that even for me port 22 fails with that cryptic error. Port 23 works.

[potemkin_ai]

@girish , thanks - the port was the thing to blame here, thanks!

One more question, if you don't mind: I can see that sub-accounts are not recommended to use with SSHFS - is still actual with Cloudron v8, or it's a relic of the past?

[girish]

@potemkin_ai We haven't tested with subaccounts in a while. If it works, you can use it. There is nothing really on Cloudron side wrt subaccounts. The docs are only having the information since we last tested it.

[potemkin_ai]

@girish yeah, I understand there is nothing about Cloudron here - just wanted to check your experience / understanding here. Thanks for sharing!