300 host-key prompts I refused to click

Title card: 300 host-key prompts I refused to click

Somewhere around the fourth site I did the maths on the back of a sticky note and didn’t like the answer.

Every site in the fleet migration gets a brand new server. Every brand new server has a host key my laptop has never seen. And the tooling doesn’t open one SSH session per site, it opens lots: provisioning checks, bootstrap, export on the old box, transfer, import on the new one, verification, the SSL step after the IP swap. Multiply that across roughly sixty sites and you land somewhere near three hundred of these:

The host key is not cached for this server:
  203.0.113.10 (port 22)
You have no guarantee that the server is the computer you think it is.
The server's ssh-ed25519 key fingerprint is:
  ssh-ed25519 255 SHA256:AbCdEf0123456789ExampleFingerprintOnly
If you trust this host, enter "y" to add the key to PuTTY's cache and carry on connecting.

Three hundred times I’d have to be sitting there. Three hundred times I’d type y without reading the fingerprint, because nobody reads the fingerprint at site forty. That’s the bit that bothered me more than the tedium, if I’m honest. A prompt you always say yes to isn’t a security control. It’s a speed bump with a guilty conscience.

(Background for anyone arriving cold: I’ve been moving a fleet of WordPress sites off Bitnami on Lightsail onto AWS’s own managed WordPress blueprint. The pillar post has the whole shape of it. The pipeline is a set of PowerShell phase scripts on a Windows laptop, and they talk to the servers through PuTTY’s plink.)

Why plink stops

plink in -batch mode won’t prompt. That’s the point of batch mode. It meets an unknown host key, refuses the connection and exits non-zero. Which is correct behaviour and completely useless to an unattended pipeline.

plink does have an answer for this: -hostkey. Hand it the fingerprint you expect and it’ll accept a server presenting that key, cache or no cache. So the problem shrinks to a smaller one. Where do I get the fingerprint from, without a human copying it off a prompt?

The obvious route, which didn’t work

My first thought was to pre-seed PuTTY’s cache. It lives in the Windows registry, so grab the key with ssh-keyscan, write it in, done.

Two things killed that. Windows’ bundled ssh-keyscan fell over talking to current OpenSSH servers, which now lead with a post-quantum key exchange it didn’t want to negotiate, and the flag you’d normally use to steer the algorithm list wouldn’t take the override. And even with the raw key in hand, PuTTY stores ed25519 keys in a format that needs elliptic-curve point decompression to produce. That’s a hundred-odd lines of careful big-integer maths to maintain forever, just to fill in a registry value.

I didn’t want to own that code. Fingerprints are strings. Strings I can pass around.

The trick: let plink tell you

When plink -batch refuses an unknown host, it doesn’t refuse quietly. It prints the whole prompt above to stderr first, fingerprint included, and then gives up.

So the provisioning script does exactly that on purpose. As soon as the new instance reports running, it fires a throwaway plink -batch at it, expects it to fail, and scrapes the fingerprint out of the error:

$deadline = (Get-Date).AddSeconds(120)
$hostKey  = ''
while ((Get-Date) -lt $deadline -and -not $hostKey) {
    $errFile = [IO.Path]::GetTempFileName()
    try {
        $probeArgs = "-batch -ssh -i `"$keyFile`" admin@$newIp echo probe"
        Start-Process -FilePath $plink -ArgumentList $probeArgs -NoNewWindow `
            -RedirectStandardOutput 'NUL' -RedirectStandardError $errFile -Wait | Out-Null
        $err = Get-Content $errFile -Raw
        $m = [regex]::Match($err, 'SHA256:[A-Za-z0-9+/=]+')
        if ($m.Success) { $hostKey = $m.Value; break }
        if ($err -match 'Connection (refused|timed out)|Network error') {
            Start-Sleep -Seconds 5    # sshd not listening yet
            continue
        }
        break                         # some other failure, don't spin
    } finally {
        Remove-Item $errFile -ErrorAction SilentlyContinue
    }
}

The loop matters. A Lightsail instance says running a little before sshd is actually listening, so the first few probes get connection refused. Retry for a couple of minutes, then give up and warn.

The fingerprint goes into the phase’s JSON summary as new_host_key, the orchestrator hands it to every later phase as a parameter, and every plink call grows a flag:

$argString = "-batch -ssh -hostkey $hostKey -i `"$keyFile`" $user@$ip `"$remoteCmd`""

The old server gets the same treatment at the start of bootstrap. After the IP swap the static IP now points at the new instance, but it’s the same machine with the same key, so the fingerprint just carries forward.

Three hundred prompts became zero.

Is that actually safe?

This is the question I’d ask if someone showed me this, so, fair enough.

What the probe does is trust-on-first-use. So does typing y. The difference is the circumstances. I captured that key seconds after creating the instance through an authenticated AWS API call, from the same laptop, over the same network. It’s the most confident I’m ever going to be about who’s on the other end, which is a lot more than you can say for me reading a hash at eleven at night.

It isn’t perfect. Someone sitting in the network path at that exact moment could hand over their own key, and I’d pin it. They could have done the same to a human clicking yes, mind you. If your provider publishes instance host keys through its API or console, reading them from there is the better version of this, because then the fingerprint comes over a channel the attacker doesn’t control.

The part that bit me later

It worked beautifully for a while. Then one site fell over at the SSL step with this:

FATAL ERROR: Host key not in manually configured list

The fingerprint I’d captured was right. It just wasn’t the only right one.

An SSH server has several host keys, usually RSA and ed25519, and client and server negotiate which one gets used. plink leans towards whatever algorithm it already has cached for that address. That static IP had a stale RSA entry in my PuTTY cache from years of logging into the old server by hand. So the probe had negotiated ed25519 and pinned that, and the later connection to the same IP negotiated RSA because of the stale cache, and presented a perfectly legitimate key that wasn’t on my list of one.

Same machine. Different fingerprint. Depending on what my laptop happened to remember.

It had a second, sillier effect. The orchestrator has a stale-state detector that checks every phase’s saved results belong to the same migration run, and it had been comparing host-key fingerprints to decide that. Re-run a phase, negotiate a different algorithm, and it cried “stale state” about a run that was completely fine. The fix for that one was easy. Compare the AWS instance name, which doesn’t change based on the mood of a key negotiation.

The proper fix for the first problem is to capture both RSA and ed25519 fingerprints up front and pass both. plink accepts -hostkey more than once and is happy if any of them match. That one’s still on my list. The workaround in the meantime is clearing the stale cache entries for the IP before the SSL phase:

$cache = 'HKCU:\Software\SimonTatham\PuTTY\SshHostKeys'
Remove-ItemProperty -Path $cache -Name "rsa2@22:$staticIp"       -ErrorAction SilentlyContinue
Remove-ItemProperty -Path $cache -Name "ssh-ed25519@22:$staticIp" -ErrorAction SilentlyContinue

What I’d tell past me

Automating a security prompt away is fine, as long as you’re honest about what the prompt was doing and you replace it with something at least as good. Most of the time the human in that loop was only adding delay.

And a host doesn’t have a fingerprint. It has a few, and which one you see depends partly on your own client’s memory. I assumed one. The server had other ideas.

Leave a Reply

Your email address will not be published. Required fields are marked *