Abstract
Last week we moved every certificate in one of our products to a new CA in 10 minutes. It was six certificates and 18 deployments. I said issuance was the part that slows down, because most CAs limit how fast you can order, and then I waved it off.
Worst case, that 10 minutes becomes a few hours waiting for limits to reset.
A reader did the math for me. Some organizations run 30,000 certificates, they claimed, and that’s common at MSPs, telcos, and government agencies. Google Trust Services takes 100 orders an hour by default. 30,000 certificates at 100 an hour is 300 hours. That’s 12.5 days, which is not a few hours.
The arithmetic is right, but I didn’t know if 30,000 certs was realistic. TL;DR: fewer than 1,000 domains have that many.
How many certificates a domain has
We run a Certificate Transparency log search that indexes the certificates in Chrome’s trusted logs. We counted every certificate that’s valid right now and grouped them by registrable domain, so api.example.com and www.example.com both count toward example.com.
348 million domains have at least one active certificate. This is how many certificates each domain has, broken out logarithmically.
| Active certificates | Domains with more | Share of domains |
|---|---|---|
| 30 | 1,985,921 | 0.57% |
| 300 | 74,986 | 0.022% |
| 3,000 | 5,526 | 0.0016% |
| 30,000 | 835 | 0.00024% |
| 300,000 | 136 | 0.00004% |
99.4% of domains have fewer than 30 certificates. These domains include tests, personal sites, and parked domains. They also include some very large companies. Berkshire Hathaway, one of the biggest companies in the US, has only three certificates for berkshirehathaway.com. Many other large organizations use wildcards certificates.
A domain isn’t a company. A big company owns dozens of domains, and an MSP’s clients bring hundreds. But Let’s Encrypt’s limits are by registered domain. It issues 50 new certificates per registered domain every week, so at least 99.4% of domains could move every certificate to Let’s Encrypt in a single burst.
Our CT log search shows the count for your own domains, and certificate discovery keeps watching them for new ones.
Who are the big guys over 30,000?
We looked at 46 of the 835 domains over 30,000. Most of the ones I could identify are platforms that issue a certificate for every customer or device. Plex gives each Plex Media Server its own certificate under plex.direct, which has 4.86 million. Azure Government, Azure China, AWS Elastic Beanstalk, Snowflake, Cloudflare, and Home Assistant’s nabu.casa are on the list too. All of those except Azure China put customer namespaces on the Public Suffix List, so to Let’s Encrypt each customer under them is a separate registered domain.
Two of the 46 are the kind of organization the reader meant, Ford and Cisco.
ford.com has 180,349 active certificates, 99.6% of them from GlobalSign. 138,054 of them are for one name, internet-venafi-canary.test.ford.com. Without it, Ford has about 42,000, and many of its biggest groups are dev and QA hosts.
cisco.com has 1.54 million, from IdenTrust. Its biggest groups are Kubernetes GitOps hosts and end-to-end test names. One of them, caas-ch05-rcdn-ocpdev-k8s-gitops.cisco.com, has 121,176 by itself.
So organizations with 30,000 public certificates do exist. They already automate issuance at scale, because nobody renews 138,054 canary certificates by hand. Moving these organizations to a different CA would be a Herculean project unto itself.
For comparison, the whole eMudhra distrust covered about 30,800 certificates across every school InCommon serves. Of the 57 large universities we checked that week, the University of Minnesota had the most still on the distrusted chain at 916.
How long does it take to migrate CAs at each size?
A CA’s limits apply to an ACME account or a project. Google Trust Services counts orders per Google Cloud project, and every ACME account in that project shares the quota. Let’s Encrypt counts per account, and per registered domain on top of that. This is how long each takes to issue new certificates at its default limits.
| Certificates on one account | Google Trust Services, one Cloud project | Let’s Encrypt, one account |
|---|---|---|
| 30 | Under a minute | Under a minute |
| 300 | About 2 hours | Under a minute |
| 3,000 | About 29 hours | About 27 hours |
| 30,000 | About 12.5 days | About 12.4 days |
Google allows 100 new orders an hour per project. Let’s Encrypt allows 300 orders per account, then one more every 36 seconds. The Let’s Encrypt column assumes no single domain needs more than 50. Past 50, a domain gets one more every 202 minutes, and that counts across every account.
At Google’s default limit, every domain with 300 certificates or fewer gets its new certificates in 2 hours or less on its own Google Cloud project. That’s 99.98% of domains, so “a few hours” holds. Let’s Encrypt is faster up to 50 certificates per domain and much slower past that. At 30,000 on one account, the reader is right. It’s 12 and a half days.
The way around it is more accounts. An MSP that gives each client its own Google Cloud project gets 100 orders an hour per client, and one client’s migration doesn’t use up another’s quota. Let’s Encrypt takes override requests, but its docs say they take a few weeks. File it before the emergency.
We document every supported CA’s limits in CA rate limits, with which ones can be raised and how to split accounts.
CertKit automates certificate lifecycle management so moving to a new CA is a configuration change.