Certificate rate limit hit during restore process?

[girish]

@infogulch Cloudron does need certs during restore operation because it uses the certs from the backup (i.e of the previous install).

The difficulty with getting multiple domains per cert is that the list of domains is not known before hand. Usually people install apps one by one... And once the list of apps is "stable", it's not a problem.

[ctrl]

@girish I logged all steps trying to reproduce the results:

0. Install OS: Ubuntu 20.04@Hetzner Cloud instance
1. Update OS: 20.04.2
2. Restore Cloudron:

cd ~/; mkdir tmp; cd $_
wget https://cloudron.io/cloudron-setup
chmod +x cloudron-setup
./cloudron-setup --provider generic --version 6.1.2
reboot

3. Open https://IP_ADDR and accept the self-signed certificate to finish setup.
4. Restore via backup config file: Getting certificate of my.example.tld ...
5. Remove old certs from /home/yellowtent/boxdata/certs
6. Tell Firefox to forget all related sites
7. Login to my.example.tld ... (Browser still uses selfsigned cert)
8. Notification center:

Failed to new certs of my.example.tld: Failed to send new order. Expecting 201, got 429 "{\n \"type\": \"urn:ietf:params:acme:error:rateLimited\",\n \"detail\": \"Error creating new order :: too many certificates already issued for exact set of domains: my.example.tld: see https://letsencrypt.org/docs/rate-limits/\",\n \"status\": 429\n}". Renewal will be retried in 12 hours

9. Status of Apps: running||restoring (status seems to vary between restore attempts)
   9.1 If status is restoring: Restore process fails due to cert/nginx error
   9.2 If status is running: Opening an app redirects to https://my.example.tld (Browser still uses selfsigned cert)
10. Reboot from within UI
11. Can't reconnect to UI: Cloudron is offline. Reconnecting.
12. journalctl -u nginx -a

Mar 08 15:17:24 hostname systemd[1]: Starting nginx - high performance web server...
Mar 08 15:17:24 hostname nginx[503]: nginx: [emerg] cannot load certificate "/home/yellowtent/boxdata/certs/example.tld.host.cert": BIO_new_file() failed (SSL: error:02001002:s>
Mar 08 15:17:24 hostname systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Mar 08 15:17:24 hostname systemd[1]: nginx.service: Failed with result 'exit-code'.
Mar 08 15:17:24 hostname systemd[1]: Failed to start nginx - high performance web server.
Mar 08 15:17:25 hostname systemd[1]: nginx.service: Scheduled restart job, restart counter is at 1.
Mar 08 15:17:25 hostname systemd[1]: Stopped nginx - high performance web server.
Mar 08 15:17:25 hostname systemd[1]: Starting nginx - high performance web server...

13. cat /var/log/nginx/error.log

2021/03/08 14:45:54 [notice] 25650#25650: signal process started
2021/03/08 14:48:33 [notice] 996#996: signal process started
2021/03/08 14:52:30 [emerg] 1174#1174: cannot load certificate "/home/yellowtent/boxdata/certs/example.tld.host.cert": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/home/yellowtent/boxdata/certs/example.tld.host.cert','r') error:2006D080:BIO routines:BIO_new_file:no such file)

[girish]

Thanks for the detailed steps.

@ctrl said in Certificate rate limit hit during restore process?:

Open https://IP_ADDR and accept the self-signed certificate to finish setup.
Restore via backup config file: Getting certificate of my.example.tld ...
Remove old certs from /home/yellowtent/boxdata/certs

Can you clarify this part. Where do you see 'Getting certificate of my.xxx` ? I assume you want through the Restore UI correct? Meaning in the Setup page, there is a Restore link at the bottom and then you uploaded the backup config. Did you see the message during the restore process. It should only say "Downloading backup.." . https://docs.cloudron.io/backups/#restore-cloudron has some screenshots of that restore link.

Finally, why did you remove old certs from /home/yellowtent/boxdata/certs. Or are you referring to removing the old certs of obsolete domains?

[ctrl]

@girish said in Certificate rate limit hit during restore process?:

Thanks for the detailed steps.

@ctrl said in Certificate rate limit hit during restore process?:

Open https://IP_ADDR and accept the self-signed certificate to finish setup.
Restore via backup config file: Getting certificate of my.example.tld ...
Remove old certs from /home/yellowtent/boxdata/certs

Can you clarify this part. Where do you see 'Getting certificate of my.xxx` ? I assume you want through the Restore UI correct? Meaning in the Setup page, there is a Restore link at the bottom and then you uploaded the backup config. Did you see the message during the restore process. It should only say "Downloading backup.." . https://docs.cloudron.io/backups/#restore-cloudron has some screenshots of that restore link.

The message appears right after I start the restore process and Cloudron notifies about the downloading process of the backup. To create a screenshot I repeated the restoring process right now. This time nginx crashed during the restore process right after the aforementioned message regarding the certs. I attached some screenshots in order. Nevertheless I am able to load the login page by force-reloading the my.xxx domain which still uses the selfsigned cert.

Finally, why did you remove old certs from /home/yellowtent/boxdata/certs. Or are you referring to removing the old certs of obsolete domains?

I just removed orphaned certs belonging to apps I tested in the past. I let all certs in place which are still associated to existing subdomains/apps we use.

[girish]

@ctrl mm, hard to make out why this happens. Are you able to send us the IP address of your server to support@, so we can try to see why it does this? You can run "cloudron-support --enable-ssh" on the server, so we can look into it. Thanks!

[ctrl]

@girish I see.
I'm going to restore the server again. As soon as it's finished I'm going to open a ticket as you requested. ^^

[girish]

I will mark this as solved since it got auto-magically solved.

[ctrl]

@girish Yes, thank you very much, I was just about to write excact the same thing.

Maybe related - since we use inwx.de as our domain registrar we have to use the Manualoption as a DNS setting. The disadvantage is we can't use a wildcard certificate in that scenario. For the most part I assume that's why we ran into the rate limit issue in first place. Maybe I'm wrong but I guess at a certain point there were just too many requests at once since there existed many orphaned certs inside ~/yellowtent/boxdata/certs from previous installed apps since 2018.

• Wouldn't it be an improvement for Manualsetups to delete the corresponding certs upon deinstallation of an app?
• Would it be possible to include inwx.de as an automated DNS provider? They offer an API.
• For the time being which registrar would be your personal recommendation for a automated DNS provider?

[mehdi]

@ctrl FYI, you don't have to change registrars: most (if not all) registrars allows you to change the DNS hosting to another service, and most DNS companies allow to use their DNS hosting for free.

[ctrl]

@mehdi
This my friend is a good point, thank you for pulling me out of the tunnel. lol