"Backup Failed" since the last Cloudron Update

girish

@Dont-Worry no worries. if you send us a mail to support@cloudron.io , we can apply the patch on your server . (We debugged the issue on somebody's server, but we cannot match forum username and support requests... If you had sent us a mail ealier, we already patched your server)

kubasbimbas

@girish You patched my server. I confirm that automatic Cloudron backup as well as app update worked correctly. Thank you for your support.

potemkin_ai

Same here on multiple servers backing up against S3. What's an ETA for the hotfix release?

girish

@potemkin_ai we will make a 8.0.4 next week .

potemkin_ai

If that's not too much, I would appreciate if you could ping this thread, once the new version released.

girish

@potemkin_ai is out now

potemkin_ai

Thank you!

potemkin_ai

@girish , Cloudron v8.0.4, backups are still failing:

Aug 28 03:00:04 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/<...>
[no timestamp]  /home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:712
[no timestamp]  resp.error = AWS.util.error(new Error(), {
[no timestamp]  ^
[no timestamp] 
[no timestamp]  The specified method is not allowed against this resource.
[no timestamp]  at Request.extractError (/home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:712:35)
[no timestamp]  at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
[no timestamp]  at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
[no timestamp]  at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:686:14)
[no timestamp]  at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)
[no timestamp]  at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)
[no timestamp]  at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10
[no timestamp]  at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9)
[no timestamp]  at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:688:12)
[no timestamp]  at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
[no timestamp]  code: 'MethodNotAllowed',
[no timestamp]  region: null,
[no timestamp]  time: 2024-08-28T01:00:04.170Z,
[no timestamp]  requestId: '45dc6280-1336-4ef2-82df-8bf7545a1d89',
[no timestamp]  extendedRequestId: '45dc6280-1336-4ef2-82df-8bf7545a1d89',
[no timestamp]  cfId: undefined,
[no timestamp]  statusCode: 405,
[no timestamp]  retryable: false,
[no timestamp]  retryDelay: 20000
[no timestamp]  }
[no timestamp] 
[no timestamp]  v20.12.2

The provider is Exoscale's SOS.

potemkin_ai

@girish , I'm sorry, is there any news on that? I still can't backup my instance.

nebulon

It seems to crash within the AWS SDK itself, I guess your S3 storage has some compatibility issues there, we have seen such problem in the past where the storage endpoint was flaky.

Maybe you can contact exoscale about this? MethodNotAllowed might indicate that they don't implement/support some required API?

potemkin_ai

@nebulon thank you! My other instance is running perfectly fine against the same endpoint and it was working fine on the same endpoint from the same instance before the update, so I'm not sure - how / why it could be an issue outside of Cloudron?

nebulon

That I don't know either, but if I interpret the error correctly the AWS SDK gets a 405 error status contacting the storage backend. Which is the HTTP error for "Method Not Allowed" so that lines up at least. I would have no clue to even start debugging this from our side to be honest. Also I don't think this is related to this initial forum thread at all. Maybe you can create a new forum thread with as much info as possible to track this.

potemkin_ai

I have provider 'Exoscale' in both cases and the backup stopped working immediately after 8.0.3, and it doesn't yet.

It seems to be an issue on topic and I really can't do match - as everything is packed inside Exoscale module of Cloudron.

nebulon

This AWS node module was updated at https://git.cloudron.io/cloudron/box/-/commit/b5fad74ea08e4c6cd06edf881612223549fb4eb3#fa288d1472d29beccb489a676f68739ad365fc47_14_14 so maybe that introduced the incompatibility with exoscale

So to be on the same page, your other Cloudron where backups are working, is on an older Cloudron version?

girish

I tested this with a test exoscale bucket in de-fra-1 region and it works. I wonder if this is IAM/Role related. I just gave it wholesale all permissions to objectstorage. Maybe you have something more specific?

potemkin_ai

@nebulon thanks.

Both boxes are 8.0.4. Both back up to Exoscale.
The only difference I can see is that the one that works points to https://sos-ch-gva-2.exo.io, and the other (that doesn't) - to https://sos-at-vie-1.exo.io.

I wouldn't expect the endpoints to be somehow different, as it would complicate operations for Exoscale, but that is really the only difference I can see.

girish

Test bucket in sos-at-vie-1.exo.io works as well . I am only testing with small backups if that matters. Does the small backup fail for you ? Like a single app? Maybe it's some multi-part stuff which is failing

potemkin_ai

@girish I tried single app backup and it produced a completely different error:

Sep 03 14:09:48 box:tasks update 4918: {"percent":30,"message":"Retrying (6) copy of ../../2024-08-06-010000-624/app_ps.domain_name_v1.17.0.tar.gz.enc. Error: IdenticalCopy: Source and copy are identical (contents and metadata). 400"}
Sep 03 14:10:08 box:tasks update 4918: {"percent":30,"message":"Retrying (7) copy of ../../2024-08-06-010000-624/app_domain_name_v4.88.3.tar.gz.enc. Error: IdenticalCopy: Source and copy are identical (contents and metadata). 400"}

It is also interesting to note, that I clicked backup on a single app and it's backing up all apps (that is just a snippet).

And - yeah - the error is different, so I guess we have a few of bunch of issues here.

girish

I see. I can't reproduce this on Exoscale SOS. Anyone else here use Exoscale?

Also, do you hit issues with other providers too or just Exoscale?

potemkin_ai

I gave up with S3 - never liked a connected to it IAM, after all, tried to switch to Hetzner's storage box - it doesn't work as well, here are the logs:

Sep 04 12:24:30 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/5383ff24-feb3-47d1-b3e8-a0001f8832a8/data/templates/backup
Sep 04 12:24:30 box:backupformat/tgz tarPack: pipeline finished: {"startTime":1725445433315,"totalSecs":37,"transferred":561899316}
Sep 04 12:24:30 box:backupupload upload completed. error: null
Sep 04 12:24:30 box:backuptask runBackupUpload: result - {"result":""}
Sep 04 12:24:30 box:backuptask uploadAppSnapshot: team.domain.com uploaded to snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8. 37.813 seconds
Sep 04 12:24:30 box:backuptask rotateAppBackup: rotating team.domain.com to path 2024-09-04-102351-773/app_team.domain.com_v1.1.1
Sep 04 12:24:30 box:tasks update 4939: {"percent":12.11111111111111,"message":"Copying /mnt/cloudronbackup/folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc to /mnt/cloudronbackup/folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc"}
Sep 04 12:24:30 box:shell copy execArgs: ssh ["-o","\"StrictHostKeyChecking no\"","-i","/home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de","-p",22,"user@user.your-storagebox.de","cp","-al","folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc","folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc"]
Sep 04 12:24:31 box:shell copy: ssh with args -o "StrictHostKeyChecking no" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc errored Error: Command failed: ssh -o "StrictHostKeyChecking no" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc
[no timestamp]  request failed on channel 0
[no timestamp] 
[no timestamp]  at genericNodeError (node:internal/errors:984:15)
[no timestamp]  at wrappedFn (node:internal/errors:538:14)
[no timestamp]  at ChildProcess.exithandler (node:child_process:422:12)
[no timestamp]  at ChildProcess.emit (node:events:518:28)
[no timestamp]  at maybeClose (node:internal/child_process:1105:16)
[no timestamp]  at ChildProcess._handle.onexit (node:internal/child_process:305:5) {
[no timestamp]  code: 255,
[no timestamp]  killed: false,
[no timestamp]  signal: null,
[no timestamp]  cmd: 'ssh -o "StrictHostKeyChecking no" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc'
[no timestamp]  }
Sep 04 12:24:31 box:backuptask copy: copied to 2024-09-04-102351-773/app_team.domain.com_v1.1.1 errored. error: SSH connection error: copy errored with code 255 message Command failed: ssh -o "StrictHostKeyChecking no" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc
[no timestamp]  request failed on channel 0
[no timestamp] 
Sep 04 12:24:31 box:taskworker Task took 39.381 seconds
Sep 04 12:24:31 box:tasks setCompleted - 4939: {"result":null,"error":{"stack":"BoxError: SSH connection error: copy errored with code 255 message Command failed: ssh -o \"StrictHostKeyChecking no\" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc\nexec request failed on channel 0\r\n\n at Object.copy (/home/yellowtent/box/src/storage/filesystem.js:180:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)","name":"BoxError","reason":"External Error","details":{},"message":"SSH connection error: copy errored with code 255 message Command failed: ssh -o \"StrictHostKeyChecking no\" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc\nexec request failed on channel 0\r\n"}}
Sep 04 12:24:31 box:tasks update 4939: {"percent":100,"result":null,"error":{"stack":"BoxError: SSH connection error: copy errored with code 255 message Command failed: ssh -o \"StrictHostKeyChecking no\" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc\nexec request failed on channel 0\r\n\n at Object.copy (/home/yellowtent/box/src/storage/filesystem.js:180:49)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)","name":"BoxError","reason":"External Error","details":{},"message":"SSH connection error: copy errored with code 255 message Command failed: ssh -o \"StrictHostKeyChecking no\" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc\nexec request failed on channel 0\r\n"}}
[no timestamp]  SSH connection error: copy errored with code 255 message Command failed: ssh -o "StrictHostKeyChecking no" -i /home/yellowtent/platformdata/sshfs/id_rsa_user.your-storagebox.de -p 22 user@user.your-storagebox.de cp -al folder_name_cloudron/snapshot/app_5383ff24-feb3-47d1-b3e8-a0001f8832a8.tar.gz.enc folder_name_cloudron/2024-09-04-102351-773/app_team.domain.com_v1.1.1.tar.gz.enc
[no timestamp]  request failed on channel 0
[no timestamp] 
[no timestamp]  at Object.copy (/home/yellowtent/box/src/storage/filesystem.js:180:49)
[no timestamp]  at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

The ssh command in question, when executed directly from the command line fails indeed with error message: exec request failed on channel 0

I've tried to allocate more memory to the backup process - just in case - but it didn't help.

Switching to rsync didn't produce any good results either - it's extremely slow and the logs are full with errors creating directory and files upload is just stalled.
SSHFS connection appears green, reconnection doesn't help.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

"Backup Failed" since the last Cloudron Update