The cutover script said everything was fine. The site was serving a self-signed certificate to every visitor. Both of those were true for eleven minutes, and the script was the last to know.
The setup
I’m moving a fleet of WordPress boxes onto a new server image, one site at a time. The cutover itself is almost boring: build the new box, load the site onto it, then swap the static IP from the old instance to the new one. DNS never changes, because the name already points at the static IP. The last phase issues a Let’s Encrypt certificate on the new box and then checks that HTTPS works.
That last check is the bit I trusted. It fetched the site’s front page, looked at the status code, and looked at who issued the certificate. If it got a 200 from a real issuer, it called the job done.
What happened
On one site the check passed. The certificate issuer was Let’s Encrypt, the status was 200, all green. Meanwhile the site’s owner messaged to say their browser was throwing a certificate warning.
I opened it myself and got the same warning. The new box was serving the blueprint’s default self-signed certificate. So why had my check passed?
Because my check hadn’t been talking to the new box.
The record I forgot existed
The static IP swap moves one thing: the IPv4 address. The instance’s IPv6 address stays on the old box. Usually nobody cares, because most domains only have an A record.
This one also had an AAAA record, pointing at the old instance’s IPv6 address. I only found that out by looking.
dig +short AAAA www.example.com # 2001:db8:abcd:12::34 <- the OLD box's IPv6
That explains both halves of the mess. Let’s Encrypt prefers IPv6 when a domain has an AAAA record, so its HTTP-01 challenge went to the old box, which knew nothing about the new token, and failed. No new certificate. The new box fell back to its self-signed default over IPv4.
And my verification curl resolved the same name through public DNS, got the same AAAA record, and connected to the old box. Which has a perfectly valid certificate. Pass.
The check wasn’t broken, exactly. It was answering a different question from the one I thought I’d asked: “does the site have working HTTPS?” instead of “does this server?”
Two fixes
The first is to stop probing through public DNS. During a migration the whole point is that DNS and the box you care about may disagree. Pin the probe to the machine it’s running on.
curl -s --resolve www.example.com:443:127.0.0.1 \
-o /dev/null -w '%{http_code}' https://www.example.com/
echo | openssl s_client -connect 127.0.0.1:443 \
-servername www.example.com 2>/dev/null \
| openssl x509 -noout -issuer
--resolve keeps the hostname (so SNI and the vhost match) but sends the connection to loopback. Now the check can only ever see the box it’s standing on. I also widened the “is this a self-signed default” test, because this blueprint’s default cert has a subject that starts with CN = ip- followed by the address, and my old pattern only knew about the Debian and Ubuntu snakeoil names. A default cert I can’t recognise as a default cert is just a certificate.
The second fix is earlier: a preflight that looks for the problem before anything moves. For each of the apex and www names, resolve AAAA and compare the answer against the source instance’s IPv6 addresses. If they match, stop and say so. The simplest remedy is to have whoever owns the DNS delete the AAAA record; IPv4 keeps working and there’s no downtime. A record pointing at Cloudflare, an IPv4-mapped address, or some third host just gets a note, since those aren’t going to bite the swap.
What I took from it
Any verification that goes out through the public front door tests whatever the front door currently points at. That’s fine on a quiet Tuesday and quietly useless in the middle of a cutover, when the front door is the one thing you know is in flux.
The eleven minutes were mostly me working out why the warning existed when my own tooling said otherwise. I’d have saved ten of them by not believing it.
