Issue with backups listings and saving backup config in 6.2
-
d19dotcareplied to girish on Mar 5, 2021, 4:04 PM last edited by d19dotca Mar 5, 2021, 4:52 PM
@girish I just noticed that when I was running into the #2 issue earlier, it's because the Region was missing. So I just re-added it in and it's starting the backup process now. I'll see if it finishes successfully and update you. It may be not an issue of the
storage
endpoint but perhaps possibly the upgrade removed the region from the configuration? I mean it was in the database query, but wasn't showing in the GUI and I could reproduce the issue until I realized the region field wasn't set so I set it and saved and no longer had the error I reported in #2 above.Edit: I reviewed the latest logs that failed earlier and see this:
2021-03-05T15:07:46.467Z box:shell backup-snapshot/app_11c6fe46-82ff-4ea4-b816-3b78a962d048 (stdout): 2021-03-05T15:07:46.465Z box:storage/s3 Error uploading [snapshot/app_11c6fe46-82ff-4ea4-b816-3b78a962d048.tar.gz]: s3 upload error. NoSuchBucket: The specified bucket does not exist. at Request.extractError (/home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:700:35) at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20) at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10) at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:688:14) at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10) at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12) at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10 at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9) at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:690:12) at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:116:18) { code: 'NoSuchBucket', region: null, time: 2021-03-05T15:07:46.464Z, requestId: 'tx4a1cd03cb20847788c1c7-0060424942', extendedRequestId: 'tx4a1cd03cb20847788c1c7-0060424942', cfId: undefined, statusCode: 404, retryable: false, retryDelay: 20000 }
Notice the Region is set to "null". I am starting to think the error here isn't so much the
storage
endpoint but something about the Region missing after the upgrade. Even though it shows in the database, it doesn't seem like it's read or something because the region is null in the logs. Or perhaps there are two issues here. -
@girish - I can confirm the latest backup was successful. So I believe the issue here isn't the
storage
endpoint at all since I'm still using it, the issue I believe was the missing region, somehow lost during the upgrade. So I think the code can be changed back to use thestorage
endpoint, and we maybe need to focus on the missing region part.Edit wait... I just realized it's using s3 suddenly, but why? I haven't updated the Cloudron app since yesterday. How did that change from storage to s3, or is that a backend change or something you can do?
-
-
@d19dotca said in Issue with backups listings and saving backup config in 6.2:
Sorry, I feel responsible for screwing this up as I reported the change from s3 to storage subdomains in OVH Object Storage
Ha ha, not at all In fact, you have been a great help helping me identify the problem and debug this. After you reported it, I tested it too, so I don't think your report was incorrect. So, I think there is some subtle difference we are missing. I wonder why the other doc page talks about s3 though.
I am going to try out the things you tried, I wonder what's happening. I can confirm though that region is important (it has to be 'de' etc. When not set, the backend defaults to us-east-1)
-
@d19dotca said in Issue with backups listings and saving backup config in 6.2:
Edit wait... I just realized it's using s3 suddenly, but why? I haven't updated the Cloudron app since yesterday. How did that change from storage to s3, or is that a backend change or something you can do?
So, just to be clear here:
- In 6.1, we used s3 subdomain
- In 6.2, I added a migration to storage subdomain - https://git.cloudron.io/cloudron/box/-/commit/e19ab45e814f4919dbc4ef18254412c4ea784387 . This one is surely wrong, but we have to figure out what is wrong about it
Also, update from 6.1 to 6.2 is not automatic since it is still unstable, so unless you manually clicked update, the Cloudron won't update from 6.1 to 6.2.
Maybe these things helps you figure what might have happened.
-
@girish Oh I mean I definitely upgraded to 6.2 earlier, and am still running it, so I'm not sure how the
storage
part that I saw yesterday has suddenly changed tos3
now. Very strange.I see I guess two issues then? I guess in this case the
storage
endpoint isn't correct since now that it's working in my environment it's usings3
(I just don't know how it changed suddenly, or was it alwayss3
still and I'm just losing my mind? lol). And the other issue appears to be losing the Region value. -
So here's what I see right now...
The endpoint seems to be s3 (which I can't explain since I'm on 6.2.0), per the screenshot here:
And to confirm I'm on 6.2.0...
So if 6.2 was supposed to change from
s3
tostorage
(and I'm sure it did when I saw it yesterday), it seems like it's somehow reverted back now which doesn't make any sense to me either. lol. So many confusing things in this one. -
@d19dotca The endpoint URLs are hardcoded in the UI. Meaning, this whole "dropdown" list of showing different storage providers etc is really just a way of setting up "Other S3 compatible" with presets (hope that explanation makes sense).
It's possible that maybe the browser did not refresh after 6.2 update. This would mean that the frontend you are using is maybe the 6.1 one and thus still using s3 subdomain. Just an idea.
-
d19dotcareplied to girish on Mar 5, 2021, 5:38 PM last edited by d19dotca Mar 5, 2021, 5:39 PM
@girish said in Issue with backups listings and saving backup config in 6.2:
It's possible that maybe the browser did not refresh after 6.2 update
I don't think that's the case, as I see this issue in two different computers with one of them running in a private browser session, so I don't think it's a cache issue in this case. Plus like I said, I swear I saw the endpoint change to
storage
after the update last night. It's only today that it seems to have changed back tos3
and I can't explain how. haha. -
I can also add that the backup I ran earlier this morning was successful still using the OVH Object Storage backend. If it's truly on version 6.2 as it appears to be, then I guess that rules out the
storage
endpoint as an issue, right? The logs also indicated a missing Region earlier from the failed backup prior.I'm not 100% sold on this being caused by the change from
s3
tostorage
endpoints yet (even though mine still showss3
despite being on 6.2). haha. I think there's something bigger afoot here, especially if we're seeing Region missing too after the upgrade.Sorry for making things confusing, just trying to update as I learn more, lol. I don't mean to be moving the goal posts.
-
@d19dotca Let's debug the main thing first. I cannot get
storage
endpoint to work. I still get the same error as you did initially "Failed Dependency The XML you provided was not well-formed or did not validate against our published schema".So, let's start with that. Why does it work for you now but it didn't previously?
-
@d19dotca OK, sorry, I can confirm you are not imagining things. If I use OVH in dropdown it does not work. If I use it in "S3 compatible", it works. (i.e testing both with
storage
subdomain) This should be easy to figure out now what is different. -
@girish lol, okay glad to know I'm not crazy. I am still wondering how it changed from
storage
tos3
as I didn't change the preset at all, it still shows OVH Object Storage, not the s3-compatible type one (though I realize they appear to use the same code in the backend). -
@d19dotca Found it! Phew.
The
storage
subdomain only supports path style API whereass3
subdomain supports subdomain style API. The path style API is already deprecated (though they have backtracked a bit on it) - https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/ . So, s3 subdomain is the way to go with it's vhost based API access.Now, for Cloudron UI, all the named providers use vhost style. This is why
storage
does not work. The 'Compat' option (and minio) uses the path style for compat reasons. -
@d19dotca It seems that apart from trying to figure why
storage
magically changed tos3
, we are good. I am suspecting maybe there is some UI bug. Just clicking around the UI to see if it's something obvious to reproduce. -
d19dotcareplied to girish on Mar 5, 2021, 6:38 PM last edited by d19dotca Mar 5, 2021, 6:38 PM
@girish Actually sorry one more thing I'm wondering about... if we believe the
storage
ands3
thing in the GUI to just be a possible GUI issue, wouldn't that mean it's in fact usingstorage
then since that's what the 6.2 code points it to and the database points to it too? And if it's indeed using thestorage
endpoint then my last backup worked, so why would that work ifstorage
isn't the right endpoint? lol Sorry just trying to wrap my head around it and make sure we've resolved it. -
@d19dotca Sure. So the db in 6.2 says
storage
and also path style disabled. This means that when a user goes to Backups view, he will seestorage
but backups will fail. Also, if you go to Backups -> Configure and Click Save making no changes, you will get that XML error.wouldn't that mean it's in fact using storage then since that's what the 6.2 code points it to and the database points to it too
Correct
And if it's indeed using the storage endpoint then my last backup worked, so why would that work if storage isn't the right endpoint?
This, I am not so sure (and I guess cause of much confusion for us). Atleast, from my tries here, I couldn't get it work with
storage
endpoint with 6.2. I don't see how it can work with path style disabled. I guess one way to know this for sure is to go to the EventLog and see if any backup succeeded after the 6.2 update but before the backup config was changed? Sadly, I think we don't log backup config in the event log. -
@d19dotca Also, looking at your post here - https://forum.cloudron.io/post/27080 . Somehow you managed to get s3 in endpoint URL. I think this is why the backup succeeded. Also, to clarify, what is in the db and what is in the UI are in "sync".
-
d19dotcareplied to girish on Mar 5, 2021, 8:16 PM last edited by d19dotca Mar 5, 2021, 8:16 PM
@girish said in Issue with backups listings and saving backup config in 6.2:
Somehow you managed to get s3 in endpoint URL. I think this is why the backup succeeded. Also, to clarify, what is in the db and what is in the UI are in "sync".
That's the thing, I didn't do that, at least not intentionally. haha. So I was about to go change to the generic s3-compatible type, but before I did that that's when I noticed the Region was empty, so I just left it as it was but set the Region and saved, then tried the backup again to see if it was successful and it was. Is it possible that saving it somehow overwrote the
storage
endpoint back tos3
? Maybe another part of the code needs to be modified?For the database, it showed the
storage
endpoint used earlier this morning, and when I run the same command now it shows thes3
endpoint instead. So I think maybe part of the confusion earlier was that when I saved the changes (where all I did was edit the Region from null to BHS), it overwrote it back tos3
fromstorage
. Do you think that's possibly what happened, if you look at the code side of it?Here's what the DB looks like now, but compare to this the link above where I pasted it this morning before I saved the change to fill in the Region field...
ubuntu@cloudron:~$ mysql -uroot -ppassword -e "SELECT * FROM box.settings WHERE name='backup_config'" mysql: [Warning] Using a password on the command line interface can be insecure. +---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | name | value | +---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | backup_config | {"provider":"ovh-objectstorage","format":"tgz","memoryLimit":4294967296,"schedulePattern":"00 00 7,19 * * *","retentionPolicy":{"keepWithinSecs":172800},"bucket":"cloudron-backups","prefix":"","accessKeyId":"<accesskey>","secretAccessKey":"<secretkey>","endpoint":"https://s3.bhs.cloud.ovh.net","region":"bhs","signatureVersion":"v4","uploadPartSize":1073741824,"encryption":null} | +---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
18/32