Abstract
Every certificate renewal automation tool has to answer one architecture question before it does anything else: where does the private key get generated? A reader who used to be “the certificate guy” at his organization emailed me this week to ask about exactly that:
Why not create the key locally and only ship the CSR back to go through the signing process? Why have the private key anywhere it doesn’t need to be?
It’s the right instinct. It’s also how most automation tools work. Certbot generates the key on the server, builds a certificate signing request, and the private key never leaves the machine.
That design is correct for one certificate on one server, but it doesn’t scale.
The ivory-tower answer
“Generate the key where it’s used” is the textbook answer, and the textbook isn’t wrong. The key never crosses the network, never sits in a backup it shouldn’t, never lives anywhere an attacker could take it from except the server itself.
But the textbook assumes a perfect, simple deployment. One server, one name, one certificate. Most real environments don’t work that way.
Did you know that Perfect Forward Secrecy made private keys a lot safer? A stolen key can’t be used to decrypt anything. It can only be used to impersonate a server, and only if the attacker can pull off a man-in-the-middle attack.
Who generates the wildcard certificate?
You can argue whether wildcards are good practice, but the practical reality is that most organizations have at least one wildcard certificate.
*.example.com is one certificate used by many hosts. If keys are only generated on the servers that use them, which server generates it? You need to pick a “master” node. Then you need to solve how the master shares the certificate and key with everyone else. Then you need to decide what happens when the master is offline during a renewal. Is a new master elected? Does renewal just fail? How do you log and monitor all of this in case it fails?
The moment a certificate is shared, the private key has to move between machines anyway. The security property that justified local generation is already gone, and now it’s gone via scp in a cron job or a shared folder.
The same SSL certificate on multiple servers
Wildcards make the problem obvious, but you don’t need one to hit it. Any high-availability setup puts the same name on multiple servers. my.trackjs.com is served by three servers behind a load balancer. They all need to present a valid certificate for the same name.
So which of the three generates it? If one does, you’re back to the master-and-distribution problem above. If all three generate their own, you now have three different certificates for the same name. Your monitoring gets noisier, verifying a renewal means checking every server individually, and you start burning through issuance rate limits.
This question comes up constantly. There’s a thread on the Let’s Encrypt community forum asking it right now, and there are hundreds like it. Everyone who scales past one server rediscovers that issuing a certificate was the easy part. Getting it where it needs to go is the part nobody solved.
The rate limit problem
If every host issues its own certificate, the requests add up faster than you’d think. Let’s Encrypt’s rate limits allow 50 certificates per registered domain every 7 days, and only 5 certificates for the exact same set of names every 7 days.
Five sounds like plenty until it isn’t. Three servers renewing independently uses three. A config change that triggers reissuance uses the rest. Now a deploy loop that recreates containers, or an engineer debugging a renewal script, locks the whole domain name out of issuance for a week. These limits are fine for small environments, and they are exactly the kind of thing a fleet of independently issuing hosts trips over.
Pick where the private key lives, on purpose
Here’s the general principle: generate the certificate in one place, and treat distribution as the real engineering problem, because it is. The alternative isn’t “keys never move.” The alternative is keys moving anyway, ad hoc, through whichever mechanism someone bolted on when the second server showed up.
This is the tradeoff we made with CertKit. The CertKit Cloud generates keys and certificates centrally, holds them encrypted, and the agent pulls them to every server that needs them, whether that’s 1 node or 10. There’s no master node to elect, no scp jobs, and one issuance per renewal regardless of fleet size.
Generating the key where it’s used is more secure in theory, but in practice, shared certificates mean keys move, and the real choice is between distribution as a designed, encrypted, audited system or distribution as tribal knowledge. Perfect forward secrecy made the key less scary to move. Operational reality made moving it unavoidable. Design for it.
CertKit handles certificate distribution, so where the private key lives is a decision, not an accident.
Comments