"Backup Failed" since the last Cloudron Update
-
@nebulon thanks.
Both boxes are 8.0.4. Both back up to Exoscale.
The only difference I can see is that the one that works points to https://sos-ch-gva-2.exo.io, and the other (that doesn't) - to https://sos-at-vie-1.exo.io.I wouldn't expect the endpoints to be somehow different, as it would complicate operations for Exoscale, but that is really the only difference I can see.
-
-
@girish I tried single app backup and it produced a completely different error:
Sep 03 14:09:48 box:tasks update 4918: {"percent":30,"message":"Retrying (6) copy of ../../2024-08-06-010000-624/app_ps.domain_name_v1.17.0.tar.gz.enc. Error: IdenticalCopy: Source and copy are identical (contents and metadata). 400"} Sep 03 14:10:08 box:tasks update 4918: {"percent":30,"message":"Retrying (7) copy of ../../2024-08-06-010000-624/app_domain_name_v4.88.3.tar.gz.enc. Error: IdenticalCopy: Source and copy are identical (contents and metadata). 400"}
It is also interesting to note, that I clicked backup on a single app and it's backing up all apps (that is just a snippet).
And - yeah - the error is different, so I guess we have a few of bunch of issues here.
-
I gave up with S3 - never liked a connected to it IAM, after all, tried to switch to Hetzner's storage box - it doesn't work as well, here are the logs:
Sep 04 12:24:30 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/5383ff24-feb3-47d1-b3e8-a0001f8832a8/data/templates/backup Sep 04 12:24:30 box:backupformat/tgz tarPack: pipeline finished: {"startTime":1725445433315,"totalSecs":37,"transferred":561899316} Sep 04 12:24:30 box:backupupload upload completed. error: null Sep 04 12:24:30 box:backuptask runBackupUpload: result - {"result":""} Sep 04 12:24:30 box:backuptask uploadAppSnapshot: team.domain.com uploaded to snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8. 37.813 seconds Sep 04 12:24:30 box:backuptask rotateAppBackup: rotating team.domain.com to path 2024-09-04-102351-773/app_team.domain.com_v1.1.1 Sep 04 12:24:30 box:tasks update 4939: {"percent":12.11111111111111,"message":"Copying /mnt/cloudronbackup/folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc to /mnt/cloudronbackup/folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc"} Sep 04 12:24:30 box:shell copy execArgs: ssh ["-o","\"StrictHostKeyChecking no\"","-i","/home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de","-p",22,"user@user.your-storagebox.de","cp","-al","folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc","folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc"] Sep 04 12:24:31 box:shell copy: ssh with args -o "StrictHostKeyChecking no" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc errored Error: Command failed: ssh -o "StrictHostKeyChecking no" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc [no timestamp] request failed on channel 0 [no timestamp] [no timestamp] at genericNodeError (node:internal/errors:984:15) [no timestamp] at wrappedFn (node:internal/errors:538:14) [no timestamp] at ChildProcess.exithandler (node:child_process:422:12) [no timestamp] at ChildProcess.emit (node:events:518:28) [no timestamp] at maybeClose (node:internal/child_process:1105:16) [no timestamp] at ChildProcess._handle.onexit (node:internal/child_process:305:5) { [no timestamp] code: 255, [no timestamp] killed: false, [no timestamp] signal: null, [no timestamp] cmd: 'ssh -o "StrictHostKeyChecking no" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc' [no timestamp] } Sep 04 12:24:31 box:backuptask copy: copied to 2024-09-04-102351-773/app_team.domain.com_v1.1.1 errored. error: SSH connection error: copy errored with code 255 message Command failed: ssh -o "StrictHostKeyChecking no" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc [no timestamp] request failed on channel 0 [no timestamp] Sep 04 12:24:31 box:taskworker Task took 39.381 seconds Sep 04 12:24:31 box:tasks setCompleted - 4939: {"result":null,"error":{"stack":"BoxError: SSH connection error: copy errored with code 255 message Command failed: ssh -o \"StrictHostKeyChecking no\" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc\nexec request failed on channel 0\r\n\n at Object.copy (/home/yellowtent/box/src/storage/filesystem.js:180:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)","name":"BoxError","reason":"External Error","details":{},"message":"SSH connection error: copy errored with code 255 message Command failed: ssh -o \"StrictHostKeyChecking no\" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc\nexec request failed on channel 0\r\n"}} Sep 04 12:24:31 box:tasks update 4939: {"percent":100,"result":null,"error":{"stack":"BoxError: SSH connection error: copy errored with code 255 message Command failed: ssh -o \"StrictHostKeyChecking no\" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc\nexec request failed on channel 0\r\n\n at Object.copy (/home/yellowtent/box/src/storage/filesystem.js:180:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)","name":"BoxError","reason":"External Error","details":{},"message":"SSH connection error: copy errored with code 255 message Command failed: ssh -o \"StrictHostKeyChecking no\" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc\nexec request failed on channel 0\r\n"}} [no timestamp] SSH connection error: copy errored with code 255 message Command failed: ssh -o "StrictHostKeyChecking no" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc [no timestamp] request failed on channel 0 [no timestamp] [no timestamp] at Object.copy (/home/yellowtent/box/src/storage/filesystem.js:180:49) [no timestamp] at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
The ssh command in question, when executed directly from the command line fails indeed with error message:
exec request failed on channel 0
I've tried to allocate more memory to the backup process - just in case - but it didn't help.
Switching to rsync didn't produce any good results either - it's extremely slow and the logs are full with errors creating directory and files upload is just stalled.
SSHFS connection appears green, reconnection doesn't help. -
@potemkin_ai said in "Backup Failed" since the last Cloudron Update:
The ssh command in question, when executed directly from the command line fails indeed with error message: exec request failed on channel 0
Can you try port 23 of Hetzner? I haven't seen this error before though.
-
@girish , thanks - the port was the thing to blame here, thanks!
One more question, if you don't mind: I can see that sub-accounts are not recommended to use with SSHFS - is still actual with Cloudron v8, or it's a relic of the past?