"Backup Failed" since the last Cloudron Update

[nebulon]

That I don't know either, but if I interpret the error correctly the AWS SDK gets a 405 error status contacting the storage backend. Which is the HTTP error for "Method Not Allowed" so that lines up at least. I would have no clue to even start debugging this from our side to be honest. Also I don't think this is related to this initial forum thread at all. Maybe you can create a new forum thread with as much info as possible to track this.

[potemkin_ai]

I have provider 'Exoscale' in both cases and the backup stopped working immediately after 8.0.3, and it doesn't yet.

It seems to be an issue on topic and I really can't do match - as everything is packed inside Exoscale module of Cloudron.

[nebulon]

This AWS node module was updated at https://git.cloudron.io/cloudron/box/-/commit/b5fad74ea08e4c6cd06edf881612223549fb4eb3#fa288d1472d29beccb489a676f68739ad365fc47_14_14 so maybe that introduced the incompatibility with exoscale

So to be on the same page, your other Cloudron where backups are working, is on an older Cloudron version?

[girish]

I tested this with a test exoscale bucket in de-fra-1 region and it works. I wonder if this is IAM/Role related. I just gave it wholesale all permissions to objectstorage. Maybe you have something more specific?

[potemkin_ai]

@nebulon thanks.

Both boxes are 8.0.4. Both back up to Exoscale.
The only difference I can see is that the one that works points to https://sos-ch-gva-2.exo.io, and the other (that doesn't) - to https://sos-at-vie-1.exo.io.

I wouldn't expect the endpoints to be somehow different, as it would complicate operations for Exoscale, but that is really the only difference I can see.

[girish]

Test bucket in sos-at-vie-1.exo.io works as well . I am only testing with small backups if that matters. Does the small backup fail for you ? Like a single app? Maybe it's some multi-part stuff which is failing

[potemkin_ai]

@girish I tried single app backup and it produced a completely different error:

Sep 03 14:09:48 box:tasks update 4918: {"percent":30,"message":"Retrying (6) copy of ../../2024-08-06-010000-624/app_ps.domain_name_v1.17.0.tar.gz.enc. Error: IdenticalCopy: Source and copy are identical (contents and metadata). 400"}
Sep 03 14:10:08 box:tasks update 4918: {"percent":30,"message":"Retrying (7) copy of ../../2024-08-06-010000-624/app_domain_name_v4.88.3.tar.gz.enc. Error: IdenticalCopy: Source and copy are identical (contents and metadata). 400"}

It is also interesting to note, that I clicked backup on a single app and it's backing up all apps (that is just a snippet).

And - yeah - the error is different, so I guess we have a few of bunch of issues here.

[girish]

I see. I can't reproduce this on Exoscale SOS. Anyone else here use Exoscale?

Also, do you hit issues with other providers too or just Exoscale?

[potemkin_ai]

I gave up with S3 - never liked a connected to it IAM, after all, tried to switch to Hetzner's storage box - it doesn't work as well, here are the logs:

Sep 04 12:24:30 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/5383ff24-feb3-47d1-b3e8-a0001f8832a8/data/templates/backup
Sep 04 12:24:30 box:backupformat/tgz tarPack: pipeline finished: {"startTime":1725445433315,"totalSecs":37,"transferred":561899316}
Sep 04 12:24:30 box:backupupload upload completed. error: null
Sep 04 12:24:30 box:backuptask runBackupUpload: result - {"result":""}
Sep 04 12:24:30 box:backuptask uploadAppSnapshot: team.domain.com uploaded to snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8. 37.813 seconds
Sep 04 12:24:30 box:backuptask rotateAppBackup: rotating team.domain.com to path 2024-09-04-102351-773/app_team.domain.com_v1.1.1
Sep 04 12:24:30 box:tasks update 4939: {"percent":12.11111111111111,"message":"Copying /mnt/cloudronbackup/folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc to /mnt/cloudronbackup/folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc"}
Sep 04 12:24:30 box:shell copy execArgs: ssh ["-o","\"StrictHostKeyChecking no\"","-i","/home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de","-p",22,"user@user.your-storagebox.de","cp","-al","folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc","folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc"]
Sep 04 12:24:31 box:shell copy: ssh with args -o "StrictHostKeyChecking no" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc errored Error: Command failed: ssh -o "StrictHostKeyChecking no" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc
[no timestamp]  request failed on channel 0
[no timestamp] 
[no timestamp]  at genericNodeError (node:internal/errors:984:15)
[no timestamp]  at wrappedFn (node:internal/errors:538:14)
[no timestamp]  at ChildProcess.exithandler (node:child_process:422:12)
[no timestamp]  at ChildProcess.emit (node:events:518:28)
[no timestamp]  at maybeClose (node:internal/child_process:1105:16)
[no timestamp]  at ChildProcess._handle.onexit (node:internal/child_process:305:5) {
[no timestamp]  code: 255,
[no timestamp]  killed: false,
[no timestamp]  signal: null,
[no timestamp]  cmd: 'ssh -o "StrictHostKeyChecking no" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc'
[no timestamp]  }
Sep 04 12:24:31 box:backuptask copy: copied to 2024-09-04-102351-773/app_team.domain.com_v1.1.1 errored. error: SSH connection error: copy errored with code 255 message Command failed: ssh -o "StrictHostKeyChecking no" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc
[no timestamp]  request failed on channel 0
[no timestamp] 
Sep 04 12:24:31 box:taskworker Task took 39.381 seconds
Sep 04 12:24:31 box:tasks setCompleted - 4939: {"result":null,"error":{"stack":"BoxError: SSH connection error: copy errored with code 255 message Command failed: ssh -o \"StrictHostKeyChecking no\" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc\nexec request failed on channel 0\r\n\n at Object.copy (/home/yellowtent/box/src/storage/filesystem.js:180:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)","name":"BoxError","reason":"External Error","details":{},"message":"SSH connection error: copy errored with code 255 message Command failed: ssh -o \"StrictHostKeyChecking no\" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc\nexec request failed on channel 0\r\n"}}
Sep 04 12:24:31 box:tasks update 4939: {"percent":100,"result":null,"error":{"stack":"BoxError: SSH connection error: copy errored with code 255 message Command failed: ssh -o \"StrictHostKeyChecking no\" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc\nexec request failed on channel 0\r\n\n at Object.copy (/home/yellowtent/box/src/storage/filesystem.js:180:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)","name":"BoxError","reason":"External Error","details":{},"message":"SSH connection error: copy errored with code 255 message Command failed: ssh -o \"StrictHostKeyChecking no\" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc\nexec request failed on channel 0\r\n"}}
[no timestamp]  SSH connection error: copy errored with code 255 message Command failed: ssh -o "StrictHostKeyChecking no" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc
[no timestamp]  request failed on channel 0
[no timestamp] 
[no timestamp]  at Object.copy (/home/yellowtent/box/src/storage/filesystem.js:180:49)
[no timestamp]  at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

The ssh command in question, when executed directly from the command line fails indeed with error message: exec request failed on channel 0

I've tried to allocate more memory to the backup process - just in case - but it didn't help.

Switching to rsync didn't produce any good results either - it's extremely slow and the logs are full with errors creating directory and files upload is just stalled.
SSHFS connection appears green, reconnection doesn't help.

[girish]

@potemkin_ai said in "Backup Failed" since the last Cloudron Update:

The ssh command in question, when executed directly from the command line fails indeed with error message: exec request failed on channel 0

Can you try port 23 of Hetzner? I haven't seen this error before though.

[girish]

I can confirm that even for me port 22 fails with that cryptic error. Port 23 works.

[potemkin_ai]

@girish , thanks - the port was the thing to blame here, thanks!

One more question, if you don't mind: I can see that sub-accounts are not recommended to use with SSHFS - is still actual with Cloudron v8, or it's a relic of the past?

[girish]

@potemkin_ai We haven't tested with subaccounts in a while. If it works, you can use it. There is nothing really on Cloudron side wrt subaccounts. The docs are only having the information since we last tested it.

[potemkin_ai]

@girish yeah, I understand there is nothing about Cloudron here - just wanted to check your experience / understanding here. Thanks for sharing!