Abstract
Three CAs had very different bad weeks this fall. eMudhra lost a root and quit. GoDaddy had to replace 403,643 certificates over a policy document. TWCA revoked 82% of its certificates over serial numbers.
TWCA and eMudhra both failed from a lack of randomness. eMudhra’s private keys had so little that anyone could compute them. TWCA’s CA certificates have serial numbers with 24 random bits where the rules require 64. That’s about 16.8 million possibilities, where the rule asks for more than 18 quintillion. The difference is what the randomness was protecting. A computable key lets anyone sign as the CA. A guessable serial number doesn’t, at least not with today’s signature hashes.
Which of the three was worst depends on what you measure. So we built the CA incident report to measure it: every TLS incident in Mozilla’s CA program since 2017, 1,581 of them across 108 CAs as of November 7, each with its severity, how many certificates it hit, what share of the CA’s certificates that was, and who found it. You can pull the data yourself as a free download.
How the three stack up
eMudhra’s is in a category of its own. Mozilla has made 11 distrust decisions since 2017, two of them eMudhra’s, and eMudhra’s keys are the only incident in the data where a CA lost control of its own private keys. eMudhra revoked the eight compromised intermediates within 14 hours, and Mozilla removed the root two days after the bug was filed. Mozilla has since set a November 16 cutoff for eMudhra’s other RSA root. eMudhra is exiting the global TLS certificate business. By size, though, it’s unremarkable. 30,800 certificates is the 43rd largest of the 786 incidents that hit at least one certificate. It also wasn’t eMudhra’s first incident. It had 11 incidents this year before the keys turned up and 7 incidents still open when they did. eMudhra only discovered 4 of its own 33 incidents.
TWCA’s hit the biggest share of its own certificates. It revoked 14 CA certificates and 52,816 certificates issued under them, 82% of its active ones. That also cleared out a second serial number problem in its customer certificates, which had 63 random bits instead of 64. TWCA met its deadlines, with the CA certificates revoked on November 5 and the rest between November 2 and 5. It also reported, on its own, unauthorized access to its registration authority server. It says it hasn’t found any misissued certificates, and its full report on both is due November 12. TWCA had 13 incidents in the nine years before October 29. It has had four since.
GoDaddy’s is the biggest by count, but it’s also the least dangerous. Its certificates carried policy identifiers and links that didn’t match its documents. Nobody got a certificate they shouldn’t have, and it was 0.26% of GoDaddy’s certificates. But it missed its deadline. GoDaddy had five days and revoked 170,566 certificates in time. It held the other 233,077 past the deadline, finishing on October 22, because revoking them over a weekend “would cause harm to the internet.”
The best and worst certificate authorities, per certificate
The best CAs for incidents tend to be the biggest. Since 2017, Let’s Encrypt has had one incident for every 21 million certificates at its peak, the most it has had in use at once. Amazon Trust Services has one per 19 million and Google Trust Services one per 13.6 million. GoDaddy is fourth at one per 4 million, October and all.
The worst CAs for incidents are small. Firmaprofesional has one incident per 55 certificates, then Netlock and Izenpe (66 each), Certinomis (69), and Certigna (180). Two of those five are Spanish and two are French. eMudhra is 26th of the 55 CAs we could rate, at one per 3,535. TWCA is 30th, at one per 5,326.
That math isn’t entirely fair on small CAs, but I don’t know how to correct for that. Most incidents are about documents, CRLs, and disclosures, and those rules cost the same at 2,000 certificates as at 700 million. A CA with a few thousand TLS certificates, that mostly issues other kinds, will look terrible per certificate. The report shows both views side by side, and only ranks CAs with a peak of at least 1,000 certificates and at least 5 incidents.
Look up your CA
Your CA is in the report. Filter to it and read what went wrong, how many certificates it hit, and who caught it.
The incident that reaches you is the one with a deadline. That’s 24 hours for the worst cases, five days for most others, or a distrust date. You can’t pick a CA that will never have one. You can make switching cheap. When we ran a distrust wargame, moving every certificate in one product to a new CA took 10 minutes. And for 99.4% of domains, issuing all the replacements takes under a minute at default CA limits. The hard part is finding every certificate and every place it’s deployed. That’s what certificate lifecycle management is for.
CertKit automates certificate lifecycle management so a CA’s bad week doesn’t become yours.