"Backup Failed" since the last Cloudron Update

[kubasbimbas]

I have noticed that the backup issue affects also the app updates, as the app is backed up before it is updated. Last night the Immich app started scheduled update 2 times from v1.68.1, but never updated to v1.69.0. I have then started the update manually this morning and it was completed.

[joseph]

@kubasbimbas that is expected, apps are always backed up before an update. if backups fails, the update will abort and go back to previous version .

[kubasbimbas]

@joseph I have verified that the app data were uploaded to the backup backend. Let's wait for tonight scheduled update to v1.69.1.

[girish]

Just looked into this. I think there is a regression in Cloudron 8. We fixed the backup cleaner to work properly but in the process it cleans up the backup entry in the database that is being created. https://git.cloudron.io/cloudron/box/-/commit/9704eefc21af313a4239e74999186ae44a43ed46 is the fix for that. @kubasbimbas can you try that one line change? You have to add that line to the file /home/yellowtent/box/src/backupcleaner.js on the server . No need to restart anything after applying the patch.

[Dont-Worry]

@girish Hi,

What do I have to do to fix it ? I don't understand what I have to do with the git link you sent.

Thanks

[girish]

@Dont-Worry no worries. if you send us a mail to support@cloudron.io , we can apply the patch on your server . (We debugged the issue on somebody's server, but we cannot match forum username and support requests... If you had sent us a mail ealier, we already patched your server)

[kubasbimbas]

@girish You patched my server. I confirm that automatic Cloudron backup as well as app update worked correctly. Thank you for your support.

[potemkin_ai]

Same here on multiple servers backing up against S3. What's an ETA for the hotfix release?

[girish]

@potemkin_ai we will make a 8.0.4 next week .

[potemkin_ai]

If that's not too much, I would appreciate if you could ping this thread, once the new version released.

[girish]

@potemkin_ai is out now

[potemkin_ai]

Thank you!

[potemkin_ai]

@girish , Cloudron v8.0.4, backups are still failing:

Aug 28 03:00:04 box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/<...>
[no timestamp]  /home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:712
[no timestamp]  resp.error = AWS.util.error(new Error(), {
[no timestamp]  ^
[no timestamp] 
[no timestamp]  The specified method is not allowed against this resource.
[no timestamp]  at Request.extractError (/home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:712:35)
[no timestamp]  at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
[no timestamp]  at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
[no timestamp]  at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:686:14)
[no timestamp]  at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)
[no timestamp]  at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)
[no timestamp]  at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10
[no timestamp]  at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9)
[no timestamp]  at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:688:12)
[no timestamp]  at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
[no timestamp]  code: 'MethodNotAllowed',
[no timestamp]  region: null,
[no timestamp]  time: 2024-08-28T01:00:04.170Z,
[no timestamp]  requestId: '45dc6280-1336-4ef2-82df-8bf7545a1d89',
[no timestamp]  extendedRequestId: '45dc6280-1336-4ef2-82df-8bf7545a1d89',
[no timestamp]  cfId: undefined,
[no timestamp]  statusCode: 405,
[no timestamp]  retryable: false,
[no timestamp]  retryDelay: 20000
[no timestamp]  }
[no timestamp] 
[no timestamp]  v20.12.2

The provider is Exoscale's SOS.

[potemkin_ai]

@girish , I'm sorry, is there any news on that? I still can't backup my instance.

[nebulon]

It seems to crash within the AWS SDK itself, I guess your S3 storage has some compatibility issues there, we have seen such problem in the past where the storage endpoint was flaky.

Maybe you can contact exoscale about this? MethodNotAllowed might indicate that they don't implement/support some required API?

[potemkin_ai]

@nebulon thank you! My other instance is running perfectly fine against the same endpoint and it was working fine on the same endpoint from the same instance before the update, so I'm not sure - how / why it could be an issue outside of Cloudron?

[nebulon]

That I don't know either, but if I interpret the error correctly the AWS SDK gets a 405 error status contacting the storage backend. Which is the HTTP error for "Method Not Allowed" so that lines up at least. I would have no clue to even start debugging this from our side to be honest. Also I don't think this is related to this initial forum thread at all. Maybe you can create a new forum thread with as much info as possible to track this.

[potemkin_ai]

I have provider 'Exoscale' in both cases and the backup stopped working immediately after 8.0.3, and it doesn't yet.

It seems to be an issue on topic and I really can't do match - as everything is packed inside Exoscale module of Cloudron.

[nebulon]

This AWS node module was updated at https://git.cloudron.io/cloudron/box/-/commit/b5fad74ea08e4c6cd06edf881612223549fb4eb3#fa288d1472d29beccb489a676f68739ad365fc47_14_14 so maybe that introduced the incompatibility with exoscale

So to be on the same page, your other Cloudron where backups are working, is on an older Cloudron version?

[girish]

I tested this with a test exoscale bucket in de-fra-1 region and it works. I wonder if this is IAM/Role related. I just gave it wholesale all permissions to objectstorage. Maybe you have something more specific?