Keeping TLS certificates under control: expiry, renewal, alarms

Certificate expiries are an interesting kind of outage: technically trivial, expensive in consequences. The shop is down, logins fail, the phone rings, and the cause is a file whose lifetime was visible on every monitoring panel for months. That is exactly why we treat certificates not as a special case but as routine operations under the same rules as everything else: automated, monitored, with a way back.

ACME turns renewal into a byproduct

The cornerstone of quiet years is automating the proof. With the ACME protocol the server obtains its own certificate: it demonstrates to the certificate authority that it controls the domain name, then pulls the result onto the machine. A client such as certbot runs the sequence: serve the challenge, collect the certificate, notify the web server, repeat next time. The decisive insight is division of labor: a program that never leads the flow itself will not be able to repair it in an emergency either. Once renewal has run through automatically twice, we treat the procedure as an operating state, not a project.

The basis of the challenge is worth naming because it shapes the architecture: the HTTP-01 variant requires a reachable file under the well-known path on port 80, the DNS-01 variant proves the claim through a record in the zone. DNS-01 can issue wildcard certificates and works where port 80 is not reachable from the authority's viewpoint, but it needs the connection to the zone API. We pick the variant per server that keeps the lowest coupling between automation and manual dependency, and we treat the API credentials with the same care as the certificates themselves.

Short runs, driven more often

Working with short-lived certificates raises the renewal frequency and lowers the blast radius. That is not a contradiction; it is the sharp edge of this practice. How many failed renewal runs a routine survives is not decided by the validity period alone: it follows from the remaining validity measured against the cadence of the automatic runs. Only late in a certificate's life, when the remaining lifetime is small, does a client such as certbot make the first renewal attempt, and every further scheduled run repeats a failed attempt for as long as the certificate stays near expiry; how many attempts can lie between the start of that phase and the expiry date follows from validity, threshold and interval together. Short lifetimes fit this picture: the automation is exercised more often and a stalled process surfaces earlier, while a long-lived certificate exercises the procedure rarely and can keep a failed automation quiet for a long time before the expiry monitor speaks. The automatic renewals therefore run on their own cadence and not on the calendar of the maintenance windows: a renewal that only swaps a few files and reloads the service does not need to interrupt live operations. We keep maintenance windows for manual interventions and test runs, and a renewal near the critical margin never waits for the next window. What we measure in turn is not only the time to expiry but the remaining margin in runs: how many scheduled renewal runs lie between the current state and the critical margin.

One detail deserves attention: renewal is only finished when the consuming service has loaded the new files. A fresh certificate nginx has not seen protects nobody. So every renewal configuration ends with a reload hook and a verification that queries the served certificate over the wire rather than admiring the file.

The last line: the expiry monitor

Despite automation we keep a second, independent control: a check that reads the expiry dates of the important endpoints from outside and alarms when the margin drops under a threshold. It does not exist to replace the automation but to catch the ways automation quietly fails: the authority may throttle access, the challenge may stumble over an administrative problem, the disk may be full. Such cases are rare, but their sole symptom is a date nobody notices until users do. We set the threshold so that two stations of the normal routine have time to fix the problem without a night shift.

The inventory matters too: a domain list that stays in sync with reality prevents the class of outages where an old name is still served but its certificate is renewed by nobody. Decommissioned hosts need proper closure, otherwise old entries keep living secretly.

What we set by default

On the servers we manage the line looks like this: ACME automation with a reload hook and over-the-wire verification, a check run on every endpoint every few weeks, an independent expiry monitor for the external view, and a maintained inventory of names. The goal is a sober state: certificates are a topic for the quarterly round, not for the night. Anyone who has built this setup once wonders how it ever worked differently.

Renewals under observation

A procedure that counts as an operating state deserves the attention operating states get: a test drive. Twice a year, one server's renewal is not just run but followed: what does the client report, how long does the challenge take, does the new certificate show over the wire? That half-hour look surfaces the quiet neglects that only turn loud in an outage, such as a challenge that has been routed through a detour nobody controls anymore. The test drive also produces the small cheat sheet for emergencies: which command, in which order, with which expected result.

The second habit concerns measurement depth: beyond the expiry date, we record when the serving certificate was last replaced. A date drifting older is the earliest sign that the automation has quietly died, and it costs one line of monitoring.

If you want your certificate landscape brought onto this footing, or existing procedures reviewed, the security hardening and audits page is the right frame. Our post on unattended-upgrades shows how we apply the same automation mindset to packages, and the piece on critical security updates puts the whole picture in context.

Sources

• Let's Encrypt documentation, https://letsencrypt.org/docs/ (accessed 2026-11-04)
• RFC 8555, Automatic Certificate Management Environment (ACME), https://www.rfc-editor.org/rfc/rfc8555.html (accessed 2026-11-04)
• Certbot documentation, https://eff-certbot.readthedocs.io/en/stable/ (accessed 2026-11-04)