Recently I wrote about hedged requests: if a model call hasn’t answered after 100 seconds, start an identical one beside it and take whichever wins. I was quite pleased with it. The tests passed. The code was tidy.
Then I looked at a real build’s logs and found it had never once done the thing it was built to do.
What the logs said
A build fires about eight calls in parallel, one per section of the document. Most come back in around a minute. In the builds I was staring at, the last four or five calls all got recorded at the same moment, and that moment was the duration of the slowest one. No hedge. No second request. The build just waited for the straggler like it always had.
If the answers were arriving in different seconds, they’d have been logged in different seconds. Something was lumping them together.
My first guess was the network. A shared connection, maybe, or calls queuing for a free slot. That’s not a daft guess: the HTTP client defaults to six connections per host and I was starting eight calls. So I raised it to sixteen. That was a sensible change and it wasn’t the bug.
Measure before you fix
What I should have done first, and did second, was add timing to every call. Two clocks per call: when my collecting loop saw the answer finish, and when the HTTP client says the transfer actually finished. Plus connection time, plus a line every 30 seconds saying which calls the batch was still waiting for.
$this->logger->info('call.completed', [
'key' => $key,
'seen_s' => round($seenAt - $batchStart, 1),
'transfer_s' => $info['total_time'],
'connect_s' => $info['connect_time'],
]);
The next build made it obvious. The transfers had finished at sensible, spread-out times. The loop had only seen them much later, all in a bunch. Fast transfers, late collection. The network was innocent. The answers had been sitting there, finished, waiting for me to come and pick them up.
That’s the signature worth remembering. If seen is far above transfer, the problem is in your loop, not your connection.
What the stream does when it’s quiet
The collecting loop streams all the in-flight responses and wakes up every five seconds, even when nothing’s arrived, so it can check whether any call has been out long enough to hedge.
while ($running) {
foreach ($http->stream($running, 5.0) as $response => $chunk) {
if ($chunk->isTimeout()) {
// check ages, maybe start a second request
continue;
}
// handle real chunks
}
break; // stream pass over, so we're done
}
See the break at the bottom? I’d assumed that when the foreach finishes, the batch is finished. With a stream, that isn’t what it means.
Symfony’s stream() has a behaviour that’s documented and easy to skim past. When a response reports an idle timeout, it’s dropped from the stream. It’s still running. It just isn’t in this stream any more, and you have to call stream() again to carry on waiting for it. So the first time a five-second poll came back empty, every outstanding response fell out, the foreach ended, my break fired, and the code below it collected whatever was left one at a time, in key order, blocking on each.
A finished answer waited behind a slower one that happened to sort earlier. And the 100-second check never ran, because the loop had left long before 100 seconds.
The fix is to keep asking
$deadline = $began + $timeoutSeconds;
$emptyPasses = 0;
while ($running) {
$sawChunk = false;
foreach ($http->stream($running, 5.0) as $response => $chunk) {
$sawChunk = true;
// ... same handling as before
}
if ($rebuild) {
continue; // a hedge was added; stream the bigger set
}
if (microtime(true) >= $deadline) {
break; // the requests' own time limit
}
$emptyPasses = $sawChunk ? 0 : $emptyPasses + 1;
if ($emptyPasses >= 3) {
break; // a pass that tells us nothing, three times running
}
}
An ended pass with requests still out is normal. Wait again. The two exits are there so a genuinely broken stream can’t spin forever: the overall deadline, and three passes in a row that produced nothing at all.
Why the tests were green
This is the part I keep thinking about. I had tests for the hedging. They passed on the broken code.
They used a hand-written fake HTTP client, and my fake’s stream() method kept timed-out responses in the stream and yielded them again on the next tick. Friendly. Polite. Nothing like the real thing, which drops them. So in the tests the loop never saw a pass end early, and the bug couldn’t happen.
I changed the fake to drop a response the moment it yields a timeout, same as the real client. Then I wrote two tests that should have existed from the start. One: answers arriving in the order fast, mid, slow while the keys sort the other way round must be collected in arrival order. Two: a stall that begins with short idle polls, nowhere near the 100 second mark, must still get hedged once it’s old enough.
Both are the kind of test that would have caught it. Both pass now.
What I’d tell myself
A test double that is more cooperative than the real thing doesn’t test your code, it tests your assumptions. When something in your stack has an odd rule, and “responses leave the stream on timeout” is an odd rule, the fake has to have the odd rule too.
And log the difference between when something finished and when you noticed. It’s two extra fields and it found this in one build.
