Let's Encrypt renewal: the three classic failures

Automatic certificate renewal has a specific personality: it works flawlessly for months, then stops, and tells nobody. The command still runs on schedule. It just exits with an error that goes into a log file nobody opens. Here are the three ways it actually breaks, and the single check that catches all three.

The three failures, in one table

The third row is the expensive one, precisely because the log looks perfect. Monitoring that reads the renewal log therefore reports all is well while your visitors are looking at a red screen.

1. The validation challenge no longer reaches your server

To confirm you control the domain, the certificate authority asks for a file at /.well-known/acme-challenge/… on your site. Your server must serve it, over plain HTTP, with no interference.

What breaks it, months after setup:

• A blanket redirect to HTTPS or to a new domain, added for perfectly good reasons, that also catches the challenge path.
• A firewall or WAF rule that starts filtering unknown paths, or blocks the addresses validation arrives from.
• A framework catch-all route that turns every unknown URL into a 404 page, including the challenge file, which now returns 404 with a nicely designed layout.
• A bot protection installed in front of the site, presenting a waiting page to any client that does not run JavaScript.

The renewal command has not changed. Its environment has. That is what makes this failure hard to attribute: nobody touched the certificate, and yet it stopped renewing.

The test is one line, from any machine:

mkdir -p /var/www/html/.well-known/acme-challenge && echo ok > /var/www/html/.well-known/acme-challenge/test
curl -sI http://example.com/.well-known/acme-challenge/test | head -n 1

The expected answer is 200. A 301 to HTTPS is already a problem in some configurations, and a 403 or a 404 always is.

2. The DNS moved

If you renew a wildcard certificate, validation happens through a DNS record instead of a file. That means an API token for your DNS provider, sitting in a config file, quietly expiring or being revoked when someone rotates credentials.

Three variants of the same story:

• The token was revoked during a security clean-up, six months earlier.
• The domain was transferred to another registrar, and nobody thought to mention it to the script.
• The name servers changed, the zone is now managed elsewhere, and the script dutifully writes its record into a zone nobody consults any more.

In all three the error is explicit in the log. Somebody has to open it, and that is the whole problem. Nothing in this article is hard to fix; every single one of these failures is a ten-minute repair once you know about it. The entire cost lies in the weeks between the failure and the moment somebody notices.

3. Renewal succeeded, deployment did not

This is the cruel one. The new certificate is on disk, dated today, perfectly valid. Your web server is still presenting the old one, because it has not been reloaded since it started.

Nothing in the renewal log looks wrong. The file is right there. And your visitors are getting the full-page security warning anyway.

The cause is almost always a missing or silent reload hook:

# The reload belongs to the renewal, not to a manual habit
--deploy-hook "systemctl reload nginx"

Two traps worth knowing. A hook that fails does not fail the renewal: it leaves a message in the log and returns. And a service that needs a full restart rather than a reload will not pick up the new file; that is common on setups that terminate TLS somewhere other than the web server, behind a proxy or a load balancer, where the certificate has to be pushed in two places.

The fourth failure, rarer and more confusing

The first three come from your own setup. This one comes from elsewhere, and it confuses people because the command is correct and the server is faultless.

The rate limit was hit. Certificate authorities cap how many certificates can be issued for a given domain in a given period. You rarely reach it in normal operation, and very easily inside a debugging loop: ten reissue attempts in one afternoon, and you are locked out for a week with an expired certificate in production. That is exactly what the dry run is for, since it consumes nothing.

The account key is gone. A server reinstall, a partial restore, a container rebuilt without its volume: the client can no longer find the account that owned the certificates, and starts again from scratch without saying so clearly.

The domain no longer resolves to you. A record changed, a migration under way, a switch-over left half finished. Validation fails because it lands on a different machine, which is accurate and has nothing to do with the certificate.

What not to do while it is broken

Do not reissue in a loop: that is how a fixable problem becomes a week-long one. Do not disable HTTPS to "get the site back", because every link on the web pointing at your https:// addresses will then break, and browsers remember the preference for a long time. And do not install a self-signed certificate as a stopgap: it produces the same red page, with the same consequences, plus a second problem to undo afterwards.

And if all you want is what the public is receiving right now, without opening a terminal, the page that reads a certificate answers that exact question from one address.

Diagnosing in three commands

certbot renew --dry-run                 # replays the whole renewal without spending quota
curl -sI http://example.com/.well-known/acme-challenge/test   # does the challenge get through
echo | openssl s_client -connect example.com:443 2>/dev/null | openssl x509 -noout -dates

The first catches failures 1 and 2. The third catches failure 3, and it is the only one that tells you what the public actually receives.

The one check that catches all three

Every failure above is invisible from the inside and obvious from the outside. So look from the outside: open a TLS connection to your own hostname and read the expiry date of the certificate you are actually served.

That single measurement covers a failed challenge, a DNS problem and a missing reload at once, because all three end in the same place: an old certificate still being presented to the public. It also has the merit of assuming nothing about your setup, which keeps it valid after a change of host or of stack.

Alert at thirty days, seven days, and one. Thirty gives you room to debug calmly; one catches the reload that never happened. A daily check is plenty, and this is a case where raising the frequency buys you nothing: an expiry date does not change from one hour to the next.

Two habits worth having

Renew with margin. At sixty days on a ninety-day certificate, a failure leaves you a month rather than an afternoon. It is the default in most clients, and one of the few defaults you should absolutely not touch.

Test the renewal, not just the certificate. A dry run exists in almost every client. Running it monthly turns a silent failure into a known one, months before it matters. And because nobody else will warn you, it is your only advance notice.

When it has already expired

Do not look for the cause first. Renew, reload, get the site back, and investigate afterwards: it is the same order as the first fifteen minutes of any incident, for the same reason, namely that a blocked site costs something every minute while you work out why.

None of this changes your ranking, incidentally: HTTPS matters far less to search engines than the panic suggests. It matters to the person who cannot open your site.