Here is a thing I built to be careful: when a domain is taken off a box, the certificate and vhost aren’t deleted straight away. The domain gets flagged, and 48 hours later, if it’s still gone, a cleanup run removes the lot. In between, you can change your mind.
It turned out the grace period had three ways to quietly stop being a grace period. Two of them were about the clock. The third was about browsers, and I hadn’t thought about it at all.
How the clock works
It’s a text file. Each line is a domain and the time the wrapper first noticed it was missing from the config:
example.com 1790400000
Every cleanup run builds a fresh version of that file from what it sees this time round, then writes it over the old one. A domain that’s still missing keeps its original timestamp. A domain that’s come back is dropped. A domain that’s been missing for 48 hours or more gets deleted.
Rewriting the whole file each run is simple, and simple means the bugs are in what you forgot to carry across.
Hole one: the cleanup that forgot everyone else
The command for taking a domain off a box is disable. Under the hood it runs cleanup for that one domain, a targeted run. And the targeted run does what the comment says: skips everything that isn’t the target.
It skipped them before collecting their entries. So the file got rewritten from only the domains the run had looked at, which was one.
Disable domain B, and domain A’s clock vanishes. A was 40 hours into its 48. The next full run sees A missing, finds no record of it, and starts the clock from zero. Eight hours from deletion to forty-eight. Nobody loses anything here, because the failure goes in the safe direction, but a safety net that quietly resets itself isn’t doing the job I thought I’d given it.
The fix carries other domains’ entries across during a targeted run, but only for domains still missing from the config. One that’s been put back in shouldn’t keep a clock.
other_first_seen=$(echo "$pending_state" | awk -v f="$filename" '$1==f {print $2; exit}')
if [ -n "$other_first_seen" ] && ! grep -qx "$filename" <<< "$active_domains"; then
new_pending_state="${new_pending_state}${filename} ${other_first_seen}"$'\n'
fi
(That grep is a here-string for a reason. See the previous post.)
Hole two: the clock that survived being re-enabled
This one goes the other way, and it’s the one that worried me.
The clock only got dropped when a cleanup run noticed the domain was active again. So take a domain off, put it back, and take it off again before any cleanup run has happened in between. The first removal’s timestamp is still in the file. The second disable finds a first-seen time from days ago, decides the 48 hours is long up, and deletes the certificate and vhost on the spot.
Re-enabling was meant to be the “I changed my mind” button. Instead it left a booby trap for the next change of mind.
enable now clears the domain’s entry itself, under the same lock the rest of the tool uses:
kept=$(awk -v d="$domain" '$1 != d' "$CLEANUP_PENDING_FILE") printf '%s\n' "$kept" > "$CLEANUP_PENDING_FILE"
Every removal after a re-enable starts a fresh 48 hours.
Hole three: the browser remembers
Every vhost the tool generates sends an HSTS header:
Strict-Transport-Security: max-age=31536000
A browser that has seen it refuses plain HTTP for that host for a year, and refuses any certificate problem with no click-through option. That’s the point of HSTS, and good for a live site.
Now take a domain off the box while it still points there. After the grace period the certificate is gone. Anyone who visited in the past year hits an error page they can’t dismiss. The 48 hours was protecting me from deleting something by accident. It wasn’t doing anything for the visitors.
So the first time a domain is flagged, its vhost now switches to max-age=0, which tells returning browsers to forget the pin. It happens again on each pending run, in case a regeneration put the old header back. enable regenerates the vhost, which puts the proper header back. And a forced cleanup, which deletes everything at once, skips the unpin and logs a warning saying so.
It only helps visitors who come back during the grace period. Someone who doesn’t return for a fortnight stays pinned until their year runs out. I wrote that in the docs, because pretending otherwise would be worse.
While I was in there I turned off includeSubDomains, which I’d had on by default. It pins every subdomain of the host as well, and a new multisite subdomain with no certificate yet was simply unreachable. It’s an opt-in setting now.
What I’d take from it
State files that get rewritten wholesale are fine until a code path produces a partial view and writes it back as the full one. Any command that looks at a subset needs to say what it does with the rest.
And a grace period only protects the thing you were thinking about. I’d been thinking about the config file. The browsers had a year-long memory of their own.