Unsolved Certificate expiry problems (perhaps related to DNS migration)

[Robin]

So as I discussed a while ago, I did a DNS migration (https://forum.cloudron.io/topic/7429/how-to-do-a-smooth-dns-migration/2) from a manually updated wildcard, to Hetzner. For the most part, this has gone smoothly, but I seem to be running into a corner case with Lets Encrypt certificates. I've started getting upcoming expiry warnings for a bunch of domains now. I tried to force renewal, which gave me the following logs (for each service on subdomain.example.com)...

Aug 15 12:01:04 box:tasks update 6255: {"percent":76,"message":"Ensuring certs of xxx.subdomain.example.com"}
Aug 15 12:01:04 box:reverseproxy ensureCertificate: xxx.subdomain.example.com certificate already exists at /home/yellowtent/platformdata/nginx/cert/_.subdomain.example.com.key
Aug 15 12:01:04 box:reverseproxy expiryDate: /home/yellowtent/platformdata/nginx/cert/_.subdomain.example.com.cert notAfter=Oct 26 11:00:56 2022 GMT daysLeft=72.04156950231481
Aug 15 12:01:04 box:reverseproxy providerMatchesSync: /home/yellowtent/platformdata/nginx/cert/_.subdomain.example.com.cert subject=CN = *.subdomain.example.com domain=*.subdomain.example.com issuer=C = US, O = Let's Encrypt, CN = R3 wildcard=true/true prod=true/true issuerMismatch=false wildcardMismatch=false match=true

This looks okay, in theory, but then at the end I see the following:

Aug 15 12:01:04 box:reverseproxy expiryDate: /home/yellowtent/platformdata/nginx/cert/xxx.subdomain.example.com.cert notAfter=Sep 3 11:00:56 2022 GMT daysLeft=19.041567395833333

And the daysLeft here seems to match up with the mail warnings I'm getting...

So they don't seem to be renewing properly... Is there something I can do to force a renewal? And is this some kind of a bug/unhandled edge case in Cloudron, perhaps caused by the DNS provider switch?

[girish]

@Robin There seems to be two certs . The one with _ is the wildcard cert. The one with xxx is the single domain cert. I guess this can happen when you switch the DNS provider to programmatic to manual/wildcard or vice-versa.

Currently, are you using programmatic DNS or manual/wildcard ? If you go to the Location view of the app and click on Save, it should use the certificate of the latest configuration.

[Robin]

@girish I switched from wildcard over to programmatic (via Hetzner). I didn't update individual apps after that change, but I guess I can try that and see what happens...

[girish]

@Robin Should ideally not have to do this individually. Renew certs should do this in the background, but clearly isn't...

[Robin]

@girish Hmm, one remaining problem... What do I do about my dashboard, which is also affected by the same problem? Would changing its location temporarily (and then switching back) fix it?

[humptydumpty]

@girish said in Certificate expiry problems (perhaps related to DNS migration):

If you go to the Location view of the app and click on Save, it should use the certificate of the latest configuration.

Is it possible to have this note mentioned in the docs and somewhere on the "Domains & Certs" page?
Thanks!

[girish]

@Robin If you delete the dashboard nginx config with the name my.xx.conf in /home/yellowtent/platformdata/nginx/applications and then systemctl restart box, it should fix things up.

[Robin]

So, I still got more cert warnings despite moving the app locations around, and removing the dashboard config, so I decided to try be a bit more brutal about it. I removed the wildcard cert:

rm /home/yellowtent/platformdata/nginx/cert/_.xxx.*

Restarted box, and everything seemed OK. Then I triggered Renew All Certs again, and ... ended up with a new wildcard cert! That doesn't make sense to me... Is there something I can look at that would explain why it wanted to create that? Or some proper way I can nuke the DNS configuration from orbit so it ends up sensible again?

[girish]

@Robin Anything in the logs? This might be a bit tricky to get to the bottom of since the code assumes things when certs and configs are missing and starts from "scratch".

[Robin]

Nothing that mentioned the _ (wildcard) cert at least, which is part of why I'm stumped. I don't know what is creating it, or where to look.