A site on one of the migrated boxes had a Let’s Encrypt certificate expire on a Monday, and by the time anyone noticed, it had already gone. Nothing in my setup said a word beforehand. No email, no alert, no failed job anyone looked at.
My renewals kick in 45 days before expiry, so for a certificate to actually lapse, renewal has to have been failing for weeks without anybody noticing. That’s not one bug. It turned out to be three, stacked, each one quite capable of doing the job alone, and all of them sitting on every box in the fleet.
How renewal is meant to work
Each box runs a small wrapper around lego, the Go ACME client. It’s installed at provision time, and the admin user’s crontab calls it twice a day. The wrapper logs what it’s doing, checks each certificate, and renews anything within 45 days of expiry over HTTP-01.
Fine on paper. Here’s what was actually happening.
Cause one: the log file
The cron job’s output on the broken box was the same line, twice a day, for more than fifteen days:
tee: /opt/cert-manager/logs/cert-manager.log: Permission denied
The wrapper pipes its output through tee into its own log. The installer had run under sudo at provision time, so the logs directory was owned by root. Cron runs as admin. Admin couldn’t write the log, tee failed, and the wrapper bailed out right there — before it ever reached the part where it renews anything.
Fifteen-plus days of a job failing at its first line (that’s what the log still held; I don’t know for certain what happened before it, though cause three below is a decent suspect), and the only record of it was in a cron log that nobody reads, because why would you read the cron log of a job that’s been fine for months.
Regular readers will notice that this is a root-owned-files problem caused by running an installer as root. Yes. I know. There’s a whole series. I’d somehow not looked outside wp-content.
Cause two: the account key
Suppose the log had been writable. It still wouldn’t have worked, because lego keeps its ACME account under its own directory:
/opt/letsencrypt/accounts/acme-v02.api.letsencrypt.org/<email>/keys/<email>.key
That was also root-owned, mode 400. When I ran a renewal by hand as admin, lego couldn’t read its own account key. And it didn’t say so. The wrapper’s complaint was that the certificate file’s modification time hadn’t changed after the renew step, which is technically true and entirely unhelpful.
Both of these get the same fix, at both ends. Provisioning now hands both trees to admin straight after the installer runs:
for TREE in /opt/cert-manager /opt/letsencrypt; do
if [ -d "$TREE" ]; then
sudo chown -R admin:admin "$TREE"
fi
done
Chown only. The modes were already right; it was just the wrong owner. And the repair script that gets pushed to boxes already in service picked up the same step, counting what it’s about to change first so a dry run tells you something:
non_admin=$(find "$TREE" \( ! -user admin -o ! -group admin \) 2>/dev/null | wc -l)
Cause three: the fix I’d already made
This is the one I’m least happy about.
Back at the start of September, a couple of other sites had renewals failing with Let’s Encrypt getting a 403 on the challenge file. The Apache error log had this:
AH00670: Options FollowSymLinks and SymLinksIfOwnerMatch are both off, so the RewriteRule directive is also forbidden due to its similar ability to circumvent directory restrictions
The vhost template gave the challenge directory Options None. The parent vhost has the usual HTTP-to-HTTPS rewrite, and Apache refuses to run a rewrite inside a directory with neither symlink option on. So it 403s the very request Let’s Encrypt needs to see over plain HTTP.
I fixed it. Fixed the template in the tool’s repo, wrote a retrofit that walked every box and patched the vhost files already deployed, swapping Options None for Options FollowSymLinks inside that one <Directory> block. Configtest, reload, done. Renewals worked again.
What I didn’t patch was the copy of the vhost generator that lives on each box. The wrapper rewrites each site’s vhost from that local generator whenever it issues or renews a certificate. So my retrofit fixed the output, and the next renewal on any box quietly regenerated the vhost from the unfixed generator and put Options None straight back. Which then made the renewal after that one fail.
It’s quite an elegant trap when you look at it. The fix holds right up until the exact moment it’s needed.
The retrofit now patches the generator too:
for g in /opt/cert-manager/generate-vhost.sh; do
[ -f "$g" ] || continue
if grep -qE '^[[:space:]]*Options[[:space:]]+None[[:space:]]*$' "$g"; then
sudo cp -a "$g" "$g.bak-$(date +%Y%m%d-%H%M%S)"
sudo sed -i -E 's|^([[:space:]]*)Options[[:space:]]+None[[:space:]]*$|\1Options FollowSymLinks|' "$g"
echo " generator patched: $g"
fi
done
No Apache reload needed for that part. Apache never reads the generator; it only reads what the generator writes.
Don’t wait for cron
Once the ownership was fixed there was still a gap. Cron runs twice a day, so a box with a certificate at zero days could sit broken for up to twelve more hours waiting for its next tick. The repair script now kicks a renewal straight away as admin, but only if it actually changed something:
sudo -u admin -H bash -lc 'cd /home/admin && ./cert-renew'
-H so admin’s home directory resolves properly, -l so the login profile (and the fleet’s umask setting) loads, which means anything the renewal creates comes out with the same permissions it would get under cron. The renew command skips anything outside its 45-day window, so running it on a healthy box costs nothing and doesn’t go near Let’s Encrypt’s rate limits.
The fleet run patched the generator on 64 boxes. One other site’s certificate was renewed during that same pass with zero days remaining. So that’s two sites, really, and one of them got lucky by a day.
What I’d take from it
Two things, and they’re not the same lesson.
First, a job that fails at its first line fails silently in a way that a job failing halfway doesn’t. All the alerting I’d have thought to add would’ve been about renewal failing. Renewal never ran. If you’ve got a cron job that matters, something other than the job itself needs to notice when it hasn’t succeeded for a while. In this case, the obvious check is the certificate’s own expiry date, read from outside the box.
Second, if something regenerates a file, patching the file is a countdown, not a fix. Find whatever writes it.