Issue with backups listings and saving backup config in 6.2

d19dotca

I recently upgraded to 6.2, and after a cleanup was run for the backups, I caught what I believe are two defects...

Backup Listings were removed completely, even though they exist in the backup location. Listings now shows "No backups have been made yet" and no backup listings appear in the app level either.
Despite the backups seemingly working when manually run, if I go to configure the Backup Location just to re-save what's already there, I get the following error: The XML you provided was not well-formed or did not validate against our published schema

I'm using OVH Object Storage, if that matters at all. I mention this because I know there was a change in name from s3 to storage, not sure if related though.

girish

@d19dotca it looks like there is some issue connecting to the backend. That seems to be the reason why it thinks that the backups are not there. Does it work if you put in "S3 compatible" ?

girish

@d19dotca actually, before you do that. Can you please do this:

mysql -uroot -ppassword -e "SELECT * FROM box.settings WHERE name='backup_config'"

Can you paste the output (removing the access keys) ? There was a migration for OVH to use new domain names. I guess something went wrong.

girish

@d19dotca Well, it turns out the storage location reported in https://forum.cloudron.io/topic/4362/possibly-needing-to-update-ovh-object-storage-url was incorrect. In that post, I wondered where I got the "s3" from (and not "storage").

That's because https://docs.ovh.com/gb/en/public-cloud/getting_started_with_the_swift_S3_API/#introduction says that s3 endpoint is endpoint_url = https://s3.<public cloud region>.cloud.ovh.net

The container URL is some openstack URL. So, I will revert the change. OVH storage is so confusing.

girish

Oh wow, what a mess.

So, the storage.<region>.cloudron.ovh.net endpoint responds to S3 requests (I am guessing it responds to any request since it's a valid endpoint for openstack) but with a NotFound . This is why the backups got removed from the database.

d19dotca

@girish Oh wow, silly OVH. I’ll raise a ticket with them so they can try to fix that. Seems really weird they’d show ‘’’storage.’’’ In the UI for the container then require uploads using a different URL. Thanks for checking that though so quickly! I’ll run that MySQL query soon for you too. Is it okay for me to just choose s3 compatible then temporarily?

d19dotca

@girish said in Issue with backups listings and saving backup config in 6.2:

Can you please do this:
mysql -uroot -ppassword -e "SELECT * FROM box.settings WHERE name='backup_config'"

Not sure if it's still needed or not but here is the output (I removed the keys though):

ubuntu@cloudron:~$ mysql -uroot -ppassword -e "SELECT * FROM box.settings WHERE name='backup_config'"
mysql: [Warning] Using a password on the command line interface can be insecure.
+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| name          | value                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| backup_config | {"provider":"ovh-objectstorage","format":"tgz","memoryLimit":4294967296,"schedulePattern":"00 00 7,19 * * *","retentionPolicy":{"keepWithinSecs":172800},"bucket":"cloudron-backups","prefix":"","accessKeyId":"<accesskey>","secretAccessKey":"<secretkey>","endpoint":"https://storage.bhs.cloud.ovh.net","region":"bhs","signatureVersion":"v4","uploadPartSize":1073741824,"encryption":null} |
+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

I'll try now to use s3-compatible option for the time being until the final 6.2 is out (or 6.2.1 or whatever it will be haha).

Sorry, I feel responsible for screwing this up as I reported the change from s3 to storage subdomains in OVH Object Storage. I was just going off the GUI in the OVH control panel, I didn't realize OVH has weirdly (possibly defectively) designed backend APIs for that. So sorry.

d19dotca

One thing that's odd is even using the s3-compatible storage method, I apparently used the storage subdomain successfully, according to the post I made last year.

d19dotca

@girish I just noticed that when I was running into the #2 issue earlier, it's because the Region was missing. So I just re-added it in and it's starting the backup process now. I'll see if it finishes successfully and update you. It may be not an issue of the storage endpoint but perhaps possibly the upgrade removed the region from the configuration? I mean it was in the database query, but wasn't showing in the GUI and I could reproduce the issue until I realized the region field wasn't set so I set it and saved and no longer had the error I reported in #2 above.

Edit: I reviewed the latest logs that failed earlier and see this:

2021-03-05T15:07:46.467Z box:shell backup-snapshot/app_11c6fe46-82ff-4ea4-b816-3b78a962d048 (stdout): 2021-03-05T15:07:46.465Z box:storage/s3 Error uploading [snapshot/app_11c6fe46-82ff-4ea4-b816-3b78a962d048.tar.gz]: s3 upload error. NoSuchBucket: The specified bucket does not exist.
    at Request.extractError (/home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:700:35)
    at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:688:14)
    at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:690:12)
    at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'NoSuchBucket',
  region: null,
  time: 2021-03-05T15:07:46.464Z,
  requestId: 'tx4a1cd03cb20847788c1c7-0060424942',
  extendedRequestId: 'tx4a1cd03cb20847788c1c7-0060424942',
  cfId: undefined,
  statusCode: 404,
  retryable: false,
  retryDelay: 20000
}

Notice the Region is set to "null". I am starting to think the error here isn't so much the storage endpoint but something about the Region missing after the upgrade. Even though it shows in the database, it doesn't seem like it's read or something because the region is null in the logs. Or perhaps there are two issues here.

d19dotca

@girish - I can confirm the latest backup was successful. So I believe the issue here isn't the storage endpoint at all since I'm still using it, the issue I believe was the missing region, somehow lost during the upgrade. So I think the code can be changed back to use the storage endpoint, and we maybe need to focus on the missing region part.

Edit wait... I just realized it's using s3 suddenly, but why? I haven't updated the Cloudron app since yesterday. How did that change from storage to s3, or is that a backend change or something you can do?

d19dotca

If I go into Configure, I see that the region is missing again...

Interestingly if I don't make any changes and just hit Save again, it doesn't throw the error as earlier, but it just spins around and never completes.

girish

@d19dotca said in Issue with backups listings and saving backup config in 6.2:

Sorry, I feel responsible for screwing this up as I reported the change from s3 to storage subdomains in OVH Object Storage

Ha ha, not at all In fact, you have been a great help helping me identify the problem and debug this. After you reported it, I tested it too, so I don't think your report was incorrect. So, I think there is some subtle difference we are missing. I wonder why the other doc page talks about s3 though.

I am going to try out the things you tried, I wonder what's happening. I can confirm though that region is important (it has to be 'de' etc. When not set, the backend defaults to us-east-1)

girish

@d19dotca said in Issue with backups listings and saving backup config in 6.2:

Edit wait... I just realized it's using s3 suddenly, but why? I haven't updated the Cloudron app since yesterday. How did that change from storage to s3, or is that a backend change or something you can do?

So, just to be clear here:

In 6.1, we used s3 subdomain
In 6.2, I added a migration to storage subdomain - https://git.cloudron.io/cloudron/box/-/commit/e19ab45e814f4919dbc4ef18254412c4ea784387 . This one is surely wrong, but we have to figure out what is wrong about it

Also, update from 6.1 to 6.2 is not automatic since it is still unstable, so unless you manually clicked update, the Cloudron won't update from 6.1 to 6.2.

Maybe these things helps you figure what might have happened.

d19dotca

@girish Oh I mean I definitely upgraded to 6.2 earlier, and am still running it, so I'm not sure how the storage part that I saw yesterday has suddenly changed to s3 now. Very strange.

I see I guess two issues then? I guess in this case the storage endpoint isn't correct since now that it's working in my environment it's using s3 (I just don't know how it changed suddenly, or was it always s3 still and I'm just losing my mind? lol). And the other issue appears to be losing the Region value.

d19dotca

So here's what I see right now...

The endpoint seems to be s3 (which I can't explain since I'm on 6.2.0), per the screenshot here:

And to confirm I'm on 6.2.0...

So if 6.2 was supposed to change from s3 to storage (and I'm sure it did when I saw it yesterday), it seems like it's somehow reverted back now which doesn't make any sense to me either. lol. So many confusing things in this one.

girish

@d19dotca The endpoint URLs are hardcoded in the UI. Meaning, this whole "dropdown" list of showing different storage providers etc is really just a way of setting up "Other S3 compatible" with presets (hope that explanation makes sense).

It's possible that maybe the browser did not refresh after 6.2 update. This would mean that the frontend you are using is maybe the 6.1 one and thus still using s3 subdomain. Just an idea.

d19dotca

@girish said in Issue with backups listings and saving backup config in 6.2:

It's possible that maybe the browser did not refresh after 6.2 update

I don't think that's the case, as I see this issue in two different computers with one of them running in a private browser session, so I don't think it's a cache issue in this case. Plus like I said, I swear I saw the endpoint change to storage after the update last night. It's only today that it seems to have changed back to s3 and I can't explain how. haha.

d19dotca

I can also add that the backup I ran earlier this morning was successful still using the OVH Object Storage backend. If it's truly on version 6.2 as it appears to be, then I guess that rules out the storage endpoint as an issue, right? The logs also indicated a missing Region earlier from the failed backup prior.

I'm not 100% sold on this being caused by the change from s3 to storage endpoints yet (even though mine still shows s3 despite being on 6.2). haha. I think there's something bigger afoot here, especially if we're seeing Region missing too after the upgrade.

Sorry for making things confusing, just trying to update as I learn more, lol. I don't mean to be moving the goal posts.

girish

@d19dotca Let's debug the main thing first. I cannot get storage endpoint to work. I still get the same error as you did initially "Failed Dependency The XML you provided was not well-formed or did not validate against our published schema".

So, let's start with that. Why does it work for you now but it didn't previously?

girish

@d19dotca OK, sorry, I can confirm you are not imagining things. If I use OVH in dropdown it does not work. If I use it in "S3 compatible", it works. (i.e testing both with storage subdomain) This should be easy to figure out now what is different.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

Issue with backups listings and saving backup config in 6.2