Certificate Lifecycle Management: Why Does a Cert Nobody Knew About Keep Taking Production Down?

The outage that always looks the same

00:04 on a Sunday. Card payments start failing. The payment gateway is up, the application servers are healthy, and the first forty minutes go into looking at the application because that is where the errors surface. Eventually someone tries the internal service to service call by hand and gets a certificate expiry error from a load balancer that was built in 2021 by a contractor. Nobody knew the certificate existed. It was issued by the internal authority, installed manually, and its expiry lived in a calendar reminder on the account of an engineer who left last spring.

Every organisation that funds a certificate management project has this story, and the details barely change. The reason is structural. Certificates are created by whoever needs one, at the moment they need it, and there is no natural point in the process where the record of that certificate joins a central inventory. A spreadsheet is not a bad tool for tracking twenty certificates. It is a completely inadequate tool for tracking two thousand, and the number of certificates in a typical estate has been growing steadily because every internal service, every container workload and every device now wants its own.

The timeline is moving against manual processes

Two published facts make this urgent rather than tidy. Public TLS certificate maximum validity, which was measured in years not long ago, now sits at 398 days, and the CA/Browser Forum has voted to reduce it in stages to 47 days by March 2029. Let's Encrypt has issued 90 day certificates for years and that model is spreading.

Do the arithmetic on your own estate. A certificate that renews annually is a calendar entry. A certificate that renews every 47 days is roughly eight renewals a year, and if you have 800 certificates that is over six thousand renewal events. No team does that by hand. The shortening validity window does not create a new problem, it converts an existing tolerable inefficiency into an operational impossibility, and it puts a date on it. Organisations that have not automated by then will be issuing certificates in a panic every six weeks.

What Venafi, Keyfactor and AppViewX do well, and where estates outgrow them

These are serious products and the discovery and policy layers are genuinely good. Venafi and Keyfactor will find certificates across your network, enforce issuance policy, hold private keys properly and integrate with the mainstream endpoints. DigiCert Trust Lifecycle Manager and Sectigo Certificate Manager bundle management with their own issuance, which is convenient if you are consolidating onto one public authority. Smallstep is excellent if your world is modern and ACME shaped.

The constraint most estates hit is not discovery, it is installation. Finding a certificate on a device is a network problem, largely solved. Renewing it, installing the new one and restarting or reloading the right service is a device specific problem, and every product supports a list of endpoints that stops somewhere short of your actual estate. The F5 appliances are supported. The Citrix estate is supported. The Java keystores on the twelve year old application server are technically supported if you install an agent, which you cannot, because the vendor support contract prohibits it. The mainframe is not supported. The industrial gateway in the plant is not supported. The bespoke device your product team ships to customers is definitely not supported, and it is the one with the largest number of certificates. The result is that a licensed platform automates perhaps two thirds of the estate and the remaining third is still a spreadsheet, still manual, and still where your outage will come from.

The second constraint is private PKI. Organisations that issue internally have their own authority hierarchy, their own templates, their own approval rules about who may request a certificate for what name, and often a requirement that certain keys never leave a hardware security module. Products model this generically. If your naming conventions carry meaning, if your approval matrix reflects business unit boundaries, or if you have a certificate template per application tier, expressing that inside a product's generic model is fighting the tool.

What a custom build actually does differently

It starts with the same inventory, built from multiple discovery sources rather than one: network scanning for what is listening, certificate transparency log monitoring for anything issued publicly for your domains including the ones a marketing agency bought without telling you, cloud provider API enumeration, and agent or API based collection from the systems that will tolerate it. Reconciling those sources into one certificate object with a known location, owner and renewal path is the deliverable.

Then it treats installation as a plugin problem rather than a supported list. Each endpoint type gets a small adapter that knows how to place a key and certificate and how to make that system pick them up: a load balancer API call, an SSH sequence writing to a keystore and issuing a graceful reload, a configuration management run, a Kubernetes secret update. The adapters for your unusual systems are the reason to build, and each one is typically days rather than weeks once the framework exists. This is the specific gap between a two thirds solution and a complete one.

It also models ownership properly. Every certificate needs a named owner and an escalation path, derived from the same rules that govern the rest of your infrastructure rather than typed in by a requester. Unowned certificates get a visible fallback queue, because a certificate nobody owns is a future outage with a scheduled date.

Renewal has to be safe, not just automatic

Automatic renewal that installs a bad certificate is worse than manual renewal, so the workflow needs the same care as any production change. Renew well ahead of expiry, not at the last moment, so a failure has room to be fixed by a person. Validate after installation by connecting to the service and checking the presented chain rather than trusting that the API returned success. Keep the previous certificate available for rollback. Alert on failure to a channel someone actually watches, with the owner named. And stage the automation: start in observe mode so the platform proves it would have renewed correctly before it is allowed to touch anything, then enable automation per endpoint class as confidence builds. Teams that switch everything on at once have one bad week and lose organisational trust in the platform for a year.

What a full build includes

• Multi source discovery: network scanning, certificate transparency monitoring, cloud APIs and agent collection, reconciled into one inventory.
• Issuance integration with your public authorities and your internal PKI, including ACME where available and each authority's own API where not.
• Endpoint adapters for installation and service reload, extensible to the systems no product supports.
• Ownership rules with a visible fallback for unowned certificates.
• Policy enforcement on key type, size, validity and permitted issuers, with exceptions recorded rather than silently allowed.
• Post installation validation by live connection, plus rollback to the previous certificate.
• Expiry forecasting and a renewal calendar showing load per week, which matters as validity periods shorten.
• Audit trail of every issuance, installation and revocation, which is what an auditor and an incident review both ask for.

What it costs and how long it takes

From the projects Digital Heroes has delivered, a first release covering discovery, inventory, ownership and alerting, plus automated renewal for two or three endpoint classes, runs $70,000 to $140,000 and ships in 10 to 16 weeks. Discovery alone usually justifies the spend, because the certificates that cause outages are the ones nobody knew about. Full automation across the estate, including private PKI integration, policy enforcement, validation and rollback, and the long tail of endpoint adapters, runs $180,000 to $400,000 phased over 6 to 12 months.

What drives cost up specifically: the number of distinct endpoint types, because each adapter is a small integration with its own failure modes. Hardware security module integration, which is exacting work with little room for approximation. Network segmentation, since discovery has to reach segments that deliberately do not talk to each other and that means distributed collectors. Regulated environments where key ceremonies and separation of duties are prescribed. And any requirement to manage certificates on devices you ship to customers, which turns this into a product engineering problem as well as an infrastructure one.

What keeps it down: sequence the endpoint adapters by certificate count. Two or three adapters usually cover most of the estate, and the exotic systems can stay manual with good alerting until the framework is proven.

When to buy instead

If your services terminate TLS at a managed cloud load balancer or a CDN, use the provider's certificate manager. It is free or nearly so, it renews automatically and there is no project here. Same answer if your estate is uniformly modern and everything speaks ACME: cert-manager in Kubernetes plus Smallstep or Let's Encrypt covers it.

If you have a large but conventional enterprise estate, evaluate Venafi or Keyfactor honestly. When the supported endpoint list genuinely covers your systems, buying is faster and cheaper than building and we would tell you so.

Build when a meaningful share of your certificates live on systems no product will automate, when your private PKI has approval and template rules that a generic model cannot express, when you need certificate management inside a product you ship, or when you have already licensed a platform and are still tracking the remainder in a spreadsheet. That last case is the common one and it is usually cheaper to build the missing third than to keep paying for the two thirds and the risk.

How to choose a developer

Ask them to describe installing a certificate on your three most awkward systems, by name. The answer separates people who have automated a real estate from people who have called an issuance API. Ask what happens when the installation succeeds but the service does not pick up the new certificate, because that silent failure is the one that still takes you down at midnight.

Ask how they handle private key material: where it is generated, whether it ever transits the platform, and how hardware security module backed keys are handled. If they are relaxed about this, stop.

Ask about discovery in segmented networks, and expect distributed collectors rather than an assumption of flat connectivity. Ask whether they will monitor certificate transparency logs, because that is how you find the certificate someone in marketing bought for a campaign subdomain.

Get code and infrastructure ownership written into the contract before kickoff. At Digital Heroes the client owns the code from the first commit. A system holding the trust fabric of your estate is the last thing you want to be renting from a supplier you cannot replace.