Certs won't renew - HELP

[stevespaw]

One of our (paid) Cloud instances is refusing to update certs. Starting a couple of days ago. A manual refresh gets a lot of errors on the log. The client cannot log in - this is a big deal.
Where do I start?
filled with lines like this:
Dec 28 13:27:10 box:cert/acme2 sendSignedRequest: using nonce GJdccAF6CFXCywo_pgyPU5yKhGI7gls_ftlnroNQuIRwVCm8sb4 for url https://acme-v02.api.letsencrypt.org/acme/chall/1970589977/451963773075/ZgIswA
Dec 28 13:27:10 box:cert/acme2 waitForChallenge: status is "invalid" "{"type":"dns-01","url":"https://acme-v02.api.letsencrypt.org/acme/chall/1970589977/451963773075/ZgIswA","status":"invalid","validated":"2024-12-28T19:23:46Z","error":{"type":"urn:ietf:params:acme:error:unauthorized","detail":"Incorrect TXT record "dU24qPdE0kcVPGtd9z6Bf1KVFhUdUsO4QBFQPIpuXFc" found at _acme-challenge.fpiw-content.media","status":403},"token":"dDgFIftTVmWxvJ7vUeVLq-iux1rxrN_1-cw8SxaXWyU"}"
Dec 28 13:27:10 box:cert/acme2 Attempt 11 failed. Will retry: Unexpected status when waiting for challenge: invalid

[andreasdueren]

@stevespaw said in Certs won't renew - HELP:

Dec 28 13:27:10 box:cert/acme2 waitForChallenge: status is "invalid" "{"type":"dns-01","url":"https://acme-v02.api.letsencrypt.org/acme/chall/1970589977/451963773075/ZgIswA","status":"invalid","validated":"2024-12-28T19:23:46Z","error":{"type":"urn:ietf:params:acme:error:unauthorized","detail":"Incorrect TXT record "dU24qPdE0kcVPGtd9z6Bf1KVFhUdUsO4QBFQPIpuXFc" found at _acme-challenge.fpiw-content.media","status":403},"token":"dDgFIftTVmWxvJ7vUeVLq-iux1rxrN_1-cw8SxaXWyU"}"
Dec 28 13:27:10 box:cert/acme2 Attempt 11 failed. Will retry: Unexpected status when waiting for challenge: invalid

Not an expert when it comes to SSL but it looks like it's refusing to issue a new certificate because there is a text record DNS entry at _acme-challenge.fpiw-content.media? Not sure why that would be an issue but a quick DNS check does yield a result there. Did you try deleting this entry and reissue?

[stevespaw]

Nothing has changed from the initial install which went just fine several months ago - this just appeared.

[andreasdueren]

@stevespaw Have you tried deleting this txt entry and reissuing the certificate?

[stevespaw]

No not yet - I have no idea how that would have happened. I currently do not have direct access to DNS records.

[andreasdueren]

@stevespaw said in Certs won't renew - HELP:

No not yet - I have no idea how that would have happened. I currently do not have direct access to DNS records.

I assume this entry wasn't cleared properly after the last certificate was issued and is now preventing you from getting a new one

[stevespaw]

OK I have access to the GoDaddy account _ have deleted the TXT record and restarted - I keep seeing the TXT record change, buy I get errors that they don't match. This is now a big issue I need to solve, but I am not very knowledgeable on LetsEncrypt. Where do I go from here?

[robi]

If you set up a new sub domain, do the certs work ok?

If so, then you can move the app from its current sub/domain to another temporary one. Validate it all works. Then move it back for a fresh set of certs.

[stevespaw]

If you set up a new sub domain, do the certs work ok?

within GoDaddy or a new app in Cloudron?

BTW even the main "my.xyz.com" is also failing. Currently 4 apps installed in Cloudron.

[robi]

@stevespaw Cloudron.. as idk how you set up your DNS. Most folks here choose to have it automated by Cloudron.

For example, if you configure GoDaddy to use CloudFlare for DNS, then you can have Cloudron effortlessly auto manage DNS entries via Cloudflare integration.

[stevespaw]

Yes we have all of our Cloudrons DNS automated - We have quite a few paid instances. This one instance is failing the automated renewal. That is the issue.

[robi]

Ok, so it's Cloudron specific, have you tried rebooting for any updates that may be needed?

[stevespaw]

yes.. I tried that 2 days ago when this started happening.

[robi]

Ugh, then it might be best to migrate the domain and app to a working Cloudron for your customer until this system can be fixed up.

This is more of a @girish specialty and he's on vacay for a bit

[stevespaw]

Ouch this is bad news. We have custom apps that directly work with API's in these cloudron apps.

[robi]

@stevespaw they'll still work, just on a different cloudron, right?

Containers for a reason.

[stevespaw]

Yes it is possible to migrate to another Cloudron, but what's to say that the SSL will work on a different cloudron to the same domain that is having issues?

[robi]

@stevespaw testing.

[nebulon]

So we have looked into this and it appears that LetsEncrypt does not see the TXT records (reporting a NXDOMAN) for those when directly talking to the GoDaddy nameservers. We have set the DNS backend to manual to fallback to the HTTP acme flow. That way the certs were refreshed fine.

Lets see if other GoDaddy users also face this, or if this is just a hiccup for those associated nameservers.

[girish]

Unfortunately, we have lost the ability to test GoDaddy since they disabled API use for all customers having < 10 domains or something. See also https://www.reddit.com/r/godaddy/comments/1chs1j8/godaddy_access_denied_via_apicall/ . If anyone can reproduce this and can give us a test set up, happy to debug further.