If a script says a domain is missing from the config and you can see it sitting right there in the config, one of you is wrong. I spent an afternoon assuming it was me.
The tool in question is a small bash wrapper that manages Let’s Encrypt certificates on a WordPress box. Part of its job is cleanup. When a domain disappears from the config list, the wrapper doesn’t delete anything straight away. It flags the domain, starts a 48-hour clock, and only removes the certificate and the Apache vhost if the domain is still gone when the clock runs out. A grace period, in other words. Sensible.
On one box with 49 domains, the cleanup log started saying this:
no longer in config - pending removal in 48h
About a domain that was very much in the config. The next run flagged a different one. Then a different one again. One to three domains per run, never the same ones twice.
The diagnostic that argued with itself
I’d added a diagnostic to that code path some weeks earlier, for exactly this sort of mood. When a domain gets flagged, the script does its own independent grep against the config file and reports what it finds. That output read:
diagnostic: direct grep DID find 'example.com' in domains.conf
So the script had decided the domain was absent, then checked again a few lines later and found it. I’d labelled that branch “transient read issue” in my head and moved on, picturing a file being rewritten at the wrong moment. A reasonable guess, and wrong. Nothing was touching the file.
What was actually going on
The membership test looked like this, and there were six copies of it:
if echo "$active_domains" | grep -qx "$filename"; then
# domain is active
fi
The script runs under set -euo pipefail. That’s the bit that matters.
grep -q is polite about one thing: it stops the moment it finds a match. It doesn’t read the rest of its input. If echo hasn’t finished writing by then, echo tries to write into a pipe with nobody on the other end, gets SIGPIPE, and dies. Exit status 141, which is 128 plus signal 13.
Without pipefail, nobody cares. A pipeline’s status is the last command’s status, and grep said “found it”. With pipefail, the pipeline reports the first non-zero status of anything in it. So grep found the domain, and the if was told the pipeline failed anyway, and took the “not found” branch.
It’s a race, so it’s occasional. The more domains on the box, the more data echo has to push and the better the odds that grep gets to a match first. Forty-nine domains was enough to be annoying; a box with five would probably never have shown it.
You can watch it happen on any machine with bash. A big list makes it near-certain:
set -o pipefail list=$(seq 1 300000) echo "$list" | grep -qx 7 echo "rc=$?"
Mine prints rc=141 every time. The match is on line 7, and echo is still busy with the other 299,993 lines. In the real script the lists are tiny and the window is narrow, which is exactly why it looked like a ghost.
What it would have done
This is the part that made me stop being amused. A false “not in config” isn’t just a log line. The cleanup path also switches the domain’s vhost to send Strict-Transport-Security: max-age=0 (more on why in another post) and starts the 48-hour clock. A domain flagged on enough runs, with the flags landing past that 48 hours, would have had its live certificate and vhost deleted. On a site that was in the config the whole time.
The clock was wandering around the fleet at random and I’d got lucky that none of them had run out.
The fix
Drop the pipe. A here-string has no writer process to kill:
if grep -qx "$filename" <<< "$active_domains"; then
# domain is active
fi
Bash feeds the string to grep itself, and nothing upstream can die of SIGPIPE. I swapped all six, including a few where it would have been harmless, because “harmless here” is a bad reason to leave a trap in a script.
After that: three clean runs in a row on the box that had been flagging one to three domains every time. Not a proof, but at that failure rate it’s about as good as a Tuesday gets.
One sting in the tail. Any vhost that had already been switched to max-age=0 by the race doesn’t get put back by a normal run. Those have to be regenerated by hand with the vhost generator. I found out because I went looking.
What I’d take from it
echo "$var" | grep -q is everywhere. It’s in half the shell snippets on the internet, and in plenty of scripts that run perfectly well because they don’t use pipefail. The moment someone adds strict mode to a script (which is good practice, and I’d do it again) every one of those becomes a coin that’s slightly weighted.
If you’ve got set -o pipefail and a grep -q or a head -1 at the end of a pipeline, go and look at what’s feeding it.