Invalid response code when fetching directory : 429

[p44]

@d19dotca said in Invalid response code when fetching directory : 429:

I receive the subject error when Cloudron is trying to renew certificates.

Invalid response code when fetching directory : 429

I’ve searched but found nothing on this so far. Any ideas what is going on with this behaviour? It’s only happening on one particular app/sub-domain so far.

I'm having same issue right now

[d19dotca]

I just had the same issue again on a different domain. Twice in a row 12 hours apart. Can’t tell if this is a Cloudron issue or a Let’s Encrypt issue. I see one domain with the original error, and a second domain with a different error (that I believe I also saw posted recently elsewhere in this forum).

PS - I see that the text is wrong too… I think it should read “failed to renew” not “failed to new”.

---

Edit: The one that was just "status: invalid" seems to have been resolved. Still receiving the original 429 error though on the other domain. It doesn't help that I can't use the UI to renew certs in 6.2.4, so I can't manually force it to get fresher logs, I am stuck waiting for 12 hours each time.

[girish]

@d19dotca I fixed the typo. As for the error itself, from what I have seen this seems to go away magically.

Looks like we have to fix our notification to not be so aggressive. Maybe we should inform user of this error only if it happens say 10 days up to expiry or something. Cloudron starts renewing certs a month in advance and it seems to make people panic...

[d19dotca]

@girish Yeah totally fair enough. It's a Let's Encrypt issue that we're comfortable will resolve itself (and it seems that's the behaviour we've seen over time too) then maybe the only change needed in Cloudron is to make it less aggressive. Perhaps start renewing 30 days in advance as it does already, but don't notify of any problems until maybe 15 or 20 days in advance? That way presumably it'd be an issue that's occurring for roughly 10 days prior repeatedly that wasn't yet resolved if we see any notifications like that? Hopefully that makes sense.

I like your idea, I think less aggressive notifications for issues we know tend to resolve themselves over time would be helpful. And more importantly it'll save a bit of time on your end in less posts haha

[d19dotca]

@girish Just found this too: https://bobcares.com/blog/lets-encrypt-new-auth-status-429-error/ - Seems like one possible cause of this is too many subdomains in Cloudron for a single domain, and Let's Encrypt limiting how many are given out for the domain each week. In which case less aggressive notifications is probably a great change to be made.

I wonder if a second improvement here could be to show the actual Let's Encrypt response in its totality, rather than stripped down in Cloudron? That may be helpful so people at least know if the issue is in Cloudron or coming from Let's Encrypt's side.

[girish]

@d19dotca On Cloudron, this seems to happen when querying the public URL - https://acme-v02.api.letsencrypt.org/directory . That links provides a directory map of URLs (instead of hardcoding the URLs it in the code base). For some reason that returns a 429. So, it's not related to cert limits or account limits.

[girish]

If i see https://tools.ietf.org/html/rfc8555#page-23, there is no 429 response code.

[robi]

I am getting this message on a few domains as well.

[girish]

Does curl https://acme-v02.api.letsencrypt.org/directory return an error on the server?

[robi]

@girish no, all come back with data.

What about setting up a time to run these curls around the same time cloudron does (and log it), which may be when LE does something on their systems.

Also shifting the time cloudron does it may be good. There's hundreds of cloudrons hammering them at midnight for even more domains

[girish]

For lack of ideas, I have added a retry now when fetching directory.

[imc67]

@girish on one Cloudron I already get for the third time in a row (so each time after 12 hours) these below errors, usually it was only one time. Since this moring (so after the second and before the third) I updated Cloudron to 6.2.7:

The error was:

-------------------------------------

Invalid response code when fetching nonce : 429

-------------------------------------


Powered by https://cloudron.io

Sent at: Mon, 29 Mar 2021 12:00:07 GMT

The error was:

-------------------------------------

Unexpected status: invalid

-------------------------------------


Powered by https://cloudron.io

Sent at: Tue, 30 Mar 2021 00:05:38 GMT

The error was:

-------------------------------------

Invalid response code when fetching nonce : 429

-------------------------------------


Powered by https://cloudron.io

Sent at: Tue, 30 Mar 2021 12:00:09 GMT

[girish]

@imc67 in the next release, we will suppress those notifications. it seems to not add much value. in any case, the solution is just to wait since cloudron will try again later. it's not a Cloudron issue, the LE service just goes up/down by nature.