SOLVED Invalid response code when fetching directory : 429

d19dotca

I receive the subject error when Cloudron is trying to renew certificates.

Invalid response code when fetching directory : 429

I’ve searched but found nothing on this so far. Any ideas what is going on with this behaviour? It’s only happening on one particular app/sub-domain so far.

girish

@d19dotca can you paste the full logs? Usually it says what the response code was.

d19dotca

@girish Yes, sorry, I meant to do that but filed it from my phone earlier so couldn't easily do that. I'm on my computer now and have found the following task logs for renewing this one particular certificate (all others are successful, only this one fails):

2021-03-12T12:00:01.717Z box:tasks 8971: {"percent":7,"message":"Renewing certs of www.staging.<subdomain>.<domain>.<tld>"}
2021-03-12T12:00:01.719Z box:domains Unable to read fallback certificates of <domain>.<tld> from disk
2021-03-12T12:00:01.724Z box:reverseproxy ensureCertificate: www.staging.<subdomain>.<domain>.<tld> certificate already exists at /home/yellowtent/boxdata/certs/www.staging.<subdomain>.<domain>.<tld>.key
2021-03-12T12:00:01.742Z box:reverseproxy isExpiringSync: /home/yellowtent/boxdata/certs/www.staging.<subdomain>.<domain>.<tld>.cert Certificate will expire 1
2021-03-12T12:00:01.742Z box:reverseproxy ensureCertificate: www.staging.<subdomain>.<domain>.<tld> cert requires renewal
2021-03-12T12:00:01.742Z box:reverseproxy ensureCertificate: getting certificate for www.staging.<subdomain>.<domain>.<tld> with options {"prod":true,"performHttpAuthorization":true,"wildcard":false,"email":"<emailAddress>"}
2021-03-12T12:00:01.743Z box:cert/acme2 getCertificate: attempt 1
2021-03-12T12:00:01.743Z box:cert/acme2 getCertificate: start acme flow for www.staging.<subdomain>.<domain>.<tld> from https://acme-v02.api.letsencrypt.org/directory
2021-03-12T12:00:02.783Z box:cert/acme2 getCertificate: using existing acme account key
2021-03-12T12:00:02.891Z box:cert/acme2 registerUser: registering user
2021-03-12T12:00:04.181Z box:cert/acme2 sendSignedRequest: using nonce 0003bACthgA3dch1bIZAplagmGDezb3NMnkqqOYbUeTlw8o for url https://acme-v02.api.letsencrypt.org/acme/new-acct
2021-03-12T12:00:04.914Z box:cert/acme2 registerUser: user registered keyid: https://acme-v02.api.letsencrypt.org/acme/acct/59537731
2021-03-12T12:00:04.914Z box:cert/acme2 updateContact: registrationUri: https://acme-v02.api.letsencrypt.org/acme/acct/59537731 email: <emailAddress>
2021-03-12T12:00:05.188Z box:cert/acme2 getCertificate: attempt 2
2021-03-12T12:00:05.188Z box:cert/acme2 getCertificate: start acme flow for www.staging.<subdomain>.<domain>.<tld> from https://acme-v02.api.letsencrypt.org/directory
2021-03-12T12:00:05.460Z box:cert/acme2 getCertificate: attempt 3
2021-03-12T12:00:05.460Z box:cert/acme2 getCertificate: start acme flow for www.staging.<subdomain>.<domain>.<tld> from https://acme-v02.api.letsencrypt.org/directory
2021-03-12T12:00:05.713Z box:reverseproxy ensureCertificate: error: Invalid response code when fetching directory : 429 cert: null
2021-03-12T12:00:05.740Z box:reverseproxy isExpiringSync: /home/yellowtent/boxdata/certs/www.staging.<subdomain>.<domain>.<tld>.cert Certificate will not expire 0
2021-03-12T12:00:05.740Z box:reverseproxy ensureCertificate: continue using existing bundle since renewal failed

girish

This is probably a temporary error. Do you see this all the time?

d19dotca

@girish I thought it might be too but it’s been having for about the last 36 hours (I’ve had three failures on it so far, with it trying every 12 hours).

msbt

just got the same error the first time on 2 different cloudrons (one is still a v6.0.0, the other a v6.2.4)

imc67

@msbt @girish just received an email from one of my 4 Cloudron Premiums:

Dear Cloudron Admin,

The certificate for my.domain.tld could not be renewed.

The Cloudron will attempt to renew the certificate every 12 hours
until the certificate expires (at which point it will switch to
using the fallback certificate).

See https://docs.cloudron.io/troubleshooting/#certificates to
double check if your server is configured correctly to obtain certificates
via Let's Encrypt.

The error was:

Invalid response code when fetching nonce : 429

girish

Do you all still see the errors still? If so, can you please write to support@ and give me access to check what might be happenning ?

I tried to debug this on one other customer's server but the problem seems to have gone away atleast for their domain. It also looks like these errors come from using Wildcard/Manual DNS (and thus http based authorization + non-wildcard certs). Is that the case for you all as well?

When debugging, I noticed that the "Renew all certs" button is br0ken Guess, we will put a fix into the next patch release.

d19dotca

@girish It seems like it recently just resolved itself too in my system, the latest renewal logs seem to indicate it was successful now and I don’t have any failure notifications today. Maybe it was a Let’s Encrypt issue then? Seems weird though.

imc67

@girish said in Invalid response code when fetching directory : 429:

Do you all still see the errors still?

It seems it solved itself, the error email was almost 24 hours ago and I don't see errors in the latest log records.

p44

@d19dotca said in Invalid response code when fetching directory : 429:

I receive the subject error when Cloudron is trying to renew certificates.

Invalid response code when fetching directory : 429

I’ve searched but found nothing on this so far. Any ideas what is going on with this behaviour? It’s only happening on one particular app/sub-domain so far.

I'm having same issue right now

d19dotca

I just had the same issue again on a different domain. Twice in a row 12 hours apart. Can’t tell if this is a Cloudron issue or a Let’s Encrypt issue. I see one domain with the original error, and a second domain with a different error (that I believe I also saw posted recently elsewhere in this forum).

PS - I see that the text is wrong too… I think it should read “failed to renew” not “failed to new”.

Edit: The one that was just "status: invalid" seems to have been resolved. Still receiving the original 429 error though on the other domain. It doesn't help that I can't use the UI to renew certs in 6.2.4, so I can't manually force it to get fresher logs, I am stuck waiting for 12 hours each time.

girish

@d19dotca I fixed the typo. As for the error itself, from what I have seen this seems to go away magically.

Looks like we have to fix our notification to not be so aggressive. Maybe we should inform user of this error only if it happens say 10 days up to expiry or something. Cloudron starts renewing certs a month in advance and it seems to make people panic...

d19dotca

@girish Yeah totally fair enough. It's a Let's Encrypt issue that we're comfortable will resolve itself (and it seems that's the behaviour we've seen over time too) then maybe the only change needed in Cloudron is to make it less aggressive. Perhaps start renewing 30 days in advance as it does already, but don't notify of any problems until maybe 15 or 20 days in advance? That way presumably it'd be an issue that's occurring for roughly 10 days prior repeatedly that wasn't yet resolved if we see any notifications like that? Hopefully that makes sense.

I like your idea, I think less aggressive notifications for issues we know tend to resolve themselves over time would be helpful. And more importantly it'll save a bit of time on your end in less posts haha

d19dotca

@girish Just found this too: https://bobcares.com/blog/lets-encrypt-new-auth-status-429-error/ - Seems like one possible cause of this is too many subdomains in Cloudron for a single domain, and Let's Encrypt limiting how many are given out for the domain each week. In which case less aggressive notifications is probably a great change to be made.

I wonder if a second improvement here could be to show the actual Let's Encrypt response in its totality, rather than stripped down in Cloudron? That may be helpful so people at least know if the issue is in Cloudron or coming from Let's Encrypt's side.

girish

@d19dotca On Cloudron, this seems to happen when querying the public URL - https://acme-v02.api.letsencrypt.org/directory . That links provides a directory map of URLs (instead of hardcoding the URLs it in the code base). For some reason that returns a 429. So, it's not related to cert limits or account limits.

girish

If i see https://tools.ietf.org/html/rfc8555#page-23, there is no 429 response code.

robi

I am getting this message on a few domains as well.

girish

Does curl https://acme-v02.api.letsencrypt.org/directory return an error on the server?

robi

@girish no, all come back with data.

What about setting up a time to run these curls around the same time cloudron does (and log it), which may be when LE does something on their systems.

Also shifting the time cloudron does it may be good. There's hundreds of cloudrons hammering them at midnight for even more domains

girish

For lack of ideas, I have added a retry now when fetching directory.

imc67

@girish on one Cloudron I already get for the third time in a row (so each time after 12 hours) these below errors, usually it was only one time. Since this moring (so after the second and before the third) I updated Cloudron to 6.2.7:

The error was:

-------------------------------------

Invalid response code when fetching nonce : 429

-------------------------------------


Powered by https://cloudron.io

Sent at: Mon, 29 Mar 2021 12:00:07 GMT

The error was:

-------------------------------------

Unexpected status: invalid

-------------------------------------


Powered by https://cloudron.io

Sent at: Tue, 30 Mar 2021 00:05:38 GMT

The error was:

-------------------------------------

Invalid response code when fetching nonce : 429

-------------------------------------


Powered by https://cloudron.io

Sent at: Tue, 30 Mar 2021 12:00:09 GMT

girish

@imc67 in the next release, we will suppress those notifications. it seems to not add much value. in any case, the solution is just to wait since cloudron will try again later. it's not a Cloudron issue, the LE service just goes up/down by nature.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

SOLVED Invalid response code when fetching directory : 429