SOLVED Certificate renewal error - Namecheap

[BrutalBirdie]

Suddenly one out of my 5 domains getting a renewal error.

The certificate for webmail.domain.tld could not be renewed.

The Cloudron will attempt to renew the certificate every 12 hours
until the certificate expires (at which point it will switch to
using the fallback certificate).

See https://docs.cloudron.io/troubleshooting/#certificates to
double check if your server is configured correctly to obtain certificates
via Let's Encrypt.

The error was:

-------------------------------------

Failed to register user. Expecting 201, got 429 undefined

Dec 04 12:56:25 box:cert/acme2 getCertificate: attempt 3
Dec 04 12:56:25 box:cert/acme2 getCertificate: start acme flow for webmail.domain.tld from https://acme-v02.api.letsencrypt.org/directory
Dec 04 12:56:25 box:cert/acme2 getCertificate: will get wildcard cert for *.domain.de
Dec 04 12:56:25 box:cert/acme2 getCertificate: using existing acme account key
Dec 04 12:56:25 box:cert/acme2 registerUser: registering user
Dec 04 12:56:26 box:cert/acme2 sendSignedRequest: using nonce 0004LVNr4OzGxHIkkFRzKY-b9W8sMfQ9beRZzEJ7ZQ4-OAc for url https://acme-v02.api.letsencrypt.org/acme/new-acct
Dec 04 12:56:26 box:cert/acme2 registerUser: user registered keyid: https://acme-v02.api.letsencrypt.org/acme/acct/80399250
Dec 04 12:56:26 box:cert/acme2 updateContact: registrationUri: https://acme-v02.api.letsencrypt.org/acme/acct/80399250 email: mail@domain.tld
Dec 04 12:56:27 box:cert/acme2 sendSignedRequest: using nonce 0004M-vJSqTiVZjEzsHA8FnnYaM87Dd2_YkfyH42VD0n3eg for url https://acme-v02.api.letsencrypt.org/acme/acct/80399250
Dec 04 12:56:28 box:cert/acme2 updateContact: contact of user updated to mail@domain.tld
Dec 04 12:56:28 box:cert/acme2 newOrder: *.domain.de
Dec 04 12:56:28 box:cert/acme2 sendSignedRequest: using nonce 0004hkr5VnBmCZeu6tEzUs_OGX0SSIAfgPcliEBHwOWJkFA for url https://acme-v02.api.letsencrypt.org/acme/new-order
Dec 04 12:56:28 box:reverseproxy ensureCertificate: error: Failed to register user. Expecting 201, got 429 undefined cert: null
Dec 04 12:56:28 box:reverseproxy isExpiringSync: /home/yellowtent/boxdata/certs/_.domain.de.cert Certificate will not expire 0
Dec 04 12:56:28 box:reverseproxy ensureCertificate: continue using existing bundle since renewal failed

That is quite odd, all my domains run on namecheap, none are expire before 2021.
All other apps and cert renewal work as intended.

[girish]

Found the issue. Some LE certs have started using R3 as the intermediary cert - https://scotthelme.co.uk/lets-encrypts-new-root-and-intermediate-certificates/ . This cert has issuer text slightly different.

@BrutalBirdie You can make this one line change - https://git.cloudron.io/cloudron/box/-/commit/3e62f1913ab05750a343c197c519d38bf17d5b3b and then systemctl restart box

[BrutalBirdie]

Could this be a rate limit problem?

Checking https://crt.sh/?q=*.domain.tld

I can see the following

hiddenid	2020-12-03	2020-12-03	2021-03-03	*.domain.tld	*.domain.tld	C=US, O=Let's Encrypt, CN=R3
hiddenid	2020-12-03	2020-12-03	2021-03-03	*.domain.tld	*.domain.tld	C=US, O=Let's Encrypt, CN=R3
hiddenid	2020-12-03	2020-12-03	2021-03-03	*.domain.tld	*.domain.tld	C=US, O=Let's Encrypt, CN=R3
hiddenid	2020-12-03	2020-12-03	2021-03-03	*.domain.tld	*.domain.tld	C=US, O=Let's Encrypt, CN=R3
hiddenid	2020-12-03	2020-12-03	2021-03-03	*.domain.tld	*.domain.tld	C=US, O=Let's Encrypt, CN=R3
hiddenid	2020-12-03	2020-12-03	2021-03-03	*.domain.tld	*.domain.tld	C=US, O=Let's Encrypt, CN=R3
hiddenid	2020-12-02	2020-12-02	2021-03-02	*.domain.tld	*.domain.tld	C=US, O=Let's Encrypt, CN=R3
hiddenid	2020-12-02	2020-12-02	2021-03-02	*.domain.tld	*.domain.tld	C=US, O=Let's Encrypt, CN=R3
hiddenid	2020-12-02	2020-12-02	2021-03-02	*.domain.tld	*.domain.tld	C=US, O=Let's Encrypt, CN=R3
hiddenid	2020-12-02	2020-12-02	2021-03-02	*.domain.tld	*.domain.tld	C=US, O=Let's Encrypt, CN=R3
hiddenid	2020-10-03	2020-10-03	2021-01-01	*.domain.tld	*.domain.tld	C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
hiddenid	2020-10-03	2020-10-03	2021-01-01	*.domain.tld	*.domain.tld	C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3

6x yesterday and 4x on before that.
Looks fishy

[nebulon]

Indeed the response code indicates a rate-limit issue and also it should probably not request certs at such a high rate, especially those seem to be wildcard certs.

The cron job as such is only triggered once every 12 hours. Is there by any chance some other script running against your Cloudron api or did you hit manual certificate renewal a few times?

[BrutalBirdie]

@nebulon I hit manual today, I think 3x times.
And no other tools run against the API that could have triggered that.
But today date is 04.12.2020 all that happend on the 03.12.2020

[girish]

@brutalbirdie I think this is related to maybe Let's Encrypt being down. Sometimes when a deployment happens, it returns the 429 code. Can you try now? Same error?

[girish]

@brutalbirdie Oh, I just hit the same issue as yours in my test Cloudron. Investigating.

[girish]

Curiously, this only happens in my test installations and none of the production instances.

[girish]

Found the issue. Some LE certs have started using R3 as the intermediary cert - https://scotthelme.co.uk/lets-encrypts-new-root-and-intermediate-certificates/ . This cert has issuer text slightly different.

@BrutalBirdie You can make this one line change - https://git.cloudron.io/cloudron/box/-/commit/3e62f1913ab05750a343c197c519d38bf17d5b3b and then systemctl restart box

[BrutalBirdie]

@girish feeling lazy, will wait for the official update