Let's Encrypt Didn't seem to auto-renew

Ropyro

Hi Everyone ---

I received an automated email on Jan 6 telling me to renew my Let's Encrypt certificate. I ignored it because it has always auto-renewed. It didn't. And, on the 13th, I found myself unable to connect to Cloudron/Apps. I don't know what to do from here. Any idea why it didn't auto-renew and what I can do to move forward?

Thanks very much!

subven

Take a look at the documentation first.

You should be able to use the dashboard with the fallback cert and hit the button for manual renewal.

Ropyro

@subven Will do. Thanks for the reply

Ropyro

@Ropyro OK, was able to find a browser that didn't completely block me from accessing my Cloudron and was able to log in and manually renew certs. All is good now. Thanks again.

humptydumpty

I ran into the same thing today and noticed some of my apps had an expired cert. I just had to go to each app's settings page > Location > Save. That did it for me.

subven

Okay guys so why is your automatic renewal not working anymore? This needs some investigation

robi

I just got a notification of this as well.. a health check down because of a cert expiry on surfer.

jk

I had a similar issue: the certificate was actually renewed (as evidenced by crt.sh), but the old one expired today, causing certificate errors on my website.

Restarting the app fixed that, but that should happen automatically after renewal, I think.

In case it matters, I was using the Surfer app (io.cloudron.surfer@5.17.8), on Cloudron v7.3.4 (Ubuntu 18.04.4 LTS)

stumitchell

I had the same issue had to restart cloudtron (incognito mode works to get you in). Then restart the apps

scooke

Wouldn't you know it, I also had a similar issue recently - went to an app and suddenly it wouldn't load. A bunch of others too. I didn't know the trick that @humptydumpty shared, so I just pressed the button on the main Domain tab on the Dashboard to renew ALL the certs, and suddenly more weren't loading. I checked the logs, and I had a domain in there that didn't have an AAAA record, and was thus stalling ALL the renewals. The thing is, I'm certain many of the other domains also don't have an AAAA record. Maybe because the one in question is an IDN? Anyways, I figured out the IPv6 address and made a new AAAA record, and voila! Everything is back to normal.

BrutalBirdie

@staff
@nebulon you remember? I had the same issue with multiple Cloudron servers.

Okay, check if your Cloudron is still running Ubuntu 18.X.
If so check the box.service for errors, if there is an error with the note of:

Jan 09 17:15:00 ubuntu-2gb-fsn1-2 systemd[1]: Reload failed for Cloudron Admin.
Jan 09 17:15:03 ubuntu-2gb-fsn1-2 sudo[30793]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 09 17:15:04 ubuntu-2gb-fsn1-2 systemd[1]: Reloading Cloudron Admin.
Jan 09 17:15:04 ubuntu-2gb-fsn1-2 sudo[30793]: pam_unix(sudo:session): session closed for user root
Jan 09 17:15:09 ubuntu-2gb-fsn1-2 systemd[30858]: box.service: Failed to execute command: No such file or directory
Jan 09 17:15:09 ubuntu-2gb-fsn1-2 systemd[30858]: box.service: Failed at step EXEC spawning /usr/bin/kill: No such file or directory
Jan 09 17:15:09 ubuntu-2gb-fsn1-2 systemd[1]: box.service: Control process exited, code=exited status=203
Jan 09 17:15:09 ubuntu-2gb-fsn1-2 systemd[1]: Reload failed for Cloudron Admin.

On Ubuntu 18 it seems there is no /usr/bin/kill just /bin/kill then also check your /home/yellowtent/platformdata/CRON_SEED
The first int is the hour of the day.
Mine was 16:8 with the 1 hour diff of wrong timezone this matched up to the box crash.

Also please check if the renew log has anything inside, for me it was total empty.

So to everyone having this issue, please report if you are using Ubuntu 18 and if so your box.service has the same error.

nebulon

@BrutalBirdie right that kill fix will be part only for next release, however we have by now also seen Cloudrons on Ubuntu 20 hitting a cert renew or reload issue. So its only part of the fix it seems.

humptydumpty

I’m on v7.3.4 (Ubuntu 22.04.1 LTS), contabo vps, automated cloudflare dns w/ no proxying enabled.

girish

If anyone with this situation, can contact us on support@cloudron.io, I think we debug this further. I check around 20 servers we have access to but they seem to updating the certs just fine. Maybe some specific cert provider is having issues.

jdaviescoates

@girish fyi I hit this recently for a Wildcard DNS on 18.04. The Gandi API ones on the same server seemed to update fine.

robi

@jdaviescoates that's a good pivoting data point @jdaviescoates , mine is a wildcard setup too. Likely something specific to that branch of code..

jdaviescoates

@robi I thought so too, but this has just happened to me on one of my Gandi LiveDNS domains on a Cloudron running on Ubuntu 20.04 too

jagan

@girish Had this issue too. Will drop a mail later today.

nj

I can confirm the issue. Certificate of other domains added to Cloudron aren't renewed. Primary domain seems to be renewed.

Some certs are due 4 days. Good thing I had alerts enabled so I got notified.

Domain provider is Wildcard. Both domain.tld and *.domain.tld point to the cloudron (since last 1-2 years).

Renew all Certs shows "Configuring apps .. or something" and the progressbar disappears.

"Show Logs" shows empty window.
Download full logs -> 1 byte empty file

girish

@nj the logs thing is fixed in 7.3.5. Can you update and check?

But there is still the underlying problem of certs not renewing sometimes with 7.3.

jordanurbs

Also having this issue for several domains on my cloudron.

Manually renewing all certs, restarting apps, deleting browser cookies, nothing is fixing it.

My cloudron is on Ubuntu 20

girish

Wanted to update this thread. We found the issue, we will make a release with a fix (7.3.6) asap.

jaschaezra

@girish Thank you! I just came to report the same issue and was delighted that already had been taking care of! Great work!

girish

7.3.6 is out now which should fix this, rolling out slowly.

humptydumpty

@girish I know updates are rolled out alphabetically but is it based on the installed subdomain (ex: rambo.domain.com) or the bare domain?

girish

@humptydumpty iirc, it's on the primary domain i.e installed subdomain.

jdaviescoates

@girish that isn't very clear!

I think it's surely based on the domain name used for my.domain.tld, no?

i.e.

my.aaaaa.tld gets updated before
my.bbbbb.tld
....
my.zzzzz.tld

That's been my experience anyway.

girish

@jdaviescoates yes, that's the primary domain in cloudron terminology

jordanurbs

@girish I've still got problems after updating.

I'm assuming a manual certificate is my only option from here

girish

@jordanurbs what problem are you facing exactly? Click on the renew all button and post the logs, please.

matix131997

Hello,

I also report a problem with the certificate having on the domain yyy.xxx.tld
I noticed that the problem is common in many browsers - Firefox, Chrome, Brave and Vivaldi on the computer - the error pops up, and on Edge there is no error. On mobile devices - there is an error on all browsers.

Feb 03 10:18:41 box:tasks update 15: {"percent":51,"message":"Ensuring certs of my.yyy.xxx.tld"}
Feb 03 10:18:41 box:reverseproxy providerMatchesSync: subject=CN = *.yyy.xxx.tld domain=*.yyy.xxx.tld issuer=C = US, O = Let's Encrypt, CN = R3 wildcard=true/true prod=true/true issuerMismatch=false wildcardMismatch=false match=true
Feb 03 10:18:41 box:reverseproxy expiryDate: subject=CN = *.yyy.xxx.tld notBefore=Feb 2 16:20:50 2023 GMT notAfter=May 3 16:20:49 2023 GMT daysLeft=89.2931378587963
Feb 03 10:18:41 box:reverseproxy needsRenewal: false. force: false
Feb 03 10:18:41 box:reverseproxy ensureCertificate: my.yyy.xxx.tld acme cert exists and is up to date

girish

@matix131997 have you tried domains -> renew all certs already?

matix131997

@girish Yes

girish

@matix131997 per the logs atleast, the certs are fine (from yesterday)

Feb 03 10:18:41 box:reverseproxy expiryDate: subject=CN = *.yyy.xxx.tld notBefore=Feb 2 16:20:50 2023 GMT notAfter=May 3 16:20:49 2023 GMT daysLeft=89.2931378587963

Have you tried clearing the browser cache? If you like, you can also send us the domain to support@cloudron.io and we can check on our end.

matix131997

@girish Yes these are the certificates issued yesterday, because I put the server back up last night to move the applications from the old server. It was fine with the certificate until this morning. At work, the certificate started failing. I did a certificate refresh several times, cleared the browser and tests on several office devices and the error continues to appear.

EDIT: Now I reinstalled Cloudron but with manual settings for the domain with a Polish provider and it works fine so far. The certificate generates and displays without error. We will see in a few hours.

matix131997

@girish
I seem to have found the cause. It is probably related to the API of the domain providers. I did a test with 3 providers.

Hetzner DNS - no problem
GoDaddy - problem
Manual (domeny.tv) - no problem

EDIT: Sorry for the edit. One more test I did I used the domain that is in GoDaddy, having my.yyy.xxx-xxx.tld for manual settings. An error appears with the certificate! I have a feeling it's a problem with GoDaddy DNS or by the "-" in the domain.

jdaviescoates

@matix131997 said in Let's Encrypt Didn't seem to auto-renew:

GoDaddy,

Sounds like yet another reason to avoid GoDaddy like the plague

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Solved Let's Encrypt Didn't seem to auto-renew