Please randomize time-of-day for certificate renewal

[mcpherrinm]

I work at Let's Encrypt, which Cloudron uses to issue certificates per its documentation. We've identified that Cloudron disproportionately sends traffic at midnight and noon UTC. I suspect this is the automated renewal noted in Cloudron's documentation.

Let's Encrypt receives disproportionately high amounts of traffic at midnight UTC in particular, as well as the top of other hours. I'm attempting to find software which hardcodes these times to ask the developers to have them renew at a random time of day.

Would it be possible to have cloudron renew at a random time of day? If so, is there an update mechanism that would allow existing clients to install this fix?

Thank you in advance,

Matthew McPherrin
Let's Encrypt Site Reliability Engineering

[nebulon]

Hi @mcpherrinm thanks for bringing this up. I guess we haven't thought Cloudron became that popular to spread load a bit over time. Currently you can see the hardcoded cron schedule for cert renewal checker at https://git.cloudron.io/cloudron/box/-/blob/master/src/cron.js#L118

We will make this randomized for the next release then.

[mcpherrinm]

Thank you for promptly addressing this!

It’s not just cloudron but many pieces of software all making the same decision to renew at midnight, which leads to uneven traffic.

[robi]

@mcpherrinm How long does the midnight spike last?

How do you enjoy tracking these down?

[mcpherrinm]

The peak of the spike is about 20 seconds long, where we are returning some errors due to overload. Traffic is elevated for a few more minutes.

Tracking down the spikes is a little fun, though sometimes quite tricky. Cloudron was straightforward because it supplies a distinct user agent, which not all clients do.

[nebulon]

Alright with https://git.cloudron.io/cloudron/box/-/commit/5b4a1e0ec12554f7a4f1099bfc94d041abfca5f5 we now generate a random hour per Cloudron which acts as a seed to spread the instances over 24h.