Splitting a big job into parallel requests feels like a free lunch right up until you notice the lunch takes as long as the slowest person at the table.
I’d split an itinerary build into a handful of model calls that run side by side. Most of them come back in about a minute. Now and then one doesn’t. It just sits there, connection open, nothing arriving, for five or six minutes, and then answers as if nothing happened. The other calls are long finished. The build waits.
Why a timeout wasn’t the answer
My first instinct was a shorter timeout. Kill the stalled call at two minutes, retry, carry on.
It’s a reasonable instinct and I’ve written about why it goes wrong before. These calls do live web searching, so a healthy one can legitimately take a long time. A timeout tight enough to catch the stalls would also shoot calls that were about to finish. And a retry after a kill starts from zero, so the guess is expensive when it’s wrong.
What I actually wanted was to stop choosing between waiting and giving up.
Run a second one
The idea is old and has a name, hedged requests. If a call hasn’t answered after a set time, don’t kill it. Start an identical second request beside it. Whichever answers first wins, and you cancel the other.
If the first call was merely slow, it might still win. If it was stalled, the second one almost certainly beats it. Either way you only wait for the faster of the two, and the cost is one extra call on the rare occasion something has gone quiet.
The threshold matters. Too low and you double your bill for no reason. I picked 100 seconds, because the normal calls finish in about 60 and the stalls are measured in minutes. That leaves a decent gap on both sides.
The awkward part is the event loop
With an HTTP client that streams several responses at once, you iterate chunks as they arrive. A stalled call produces no chunks, which is the whole problem. So the loop needs a way to wake up when nothing is happening.
Symfony’s HTTP client will do that if you give stream() a timeout. It yields an idle-timeout chunk rather than throwing, and each one is a chance to look at the clock. Here’s the shape of it, trimmed right down:
while ($running) {
foreach ($http->stream($running, 5.0) as $response => $chunk) {
if ($chunk->isTimeout()) {
foreach ($started as $key => $t0) {
if (isset($hedged[$key]) || isset($done[$key])) {
continue;
}
if (microtime(true) - $t0 >= 100) {
$hedged[$key] = true;
$second = $http->request('POST', $url, $requests[$key]);
$owner[spl_object_id($second)] = $key;
$running[spl_object_id($second)] = $second;
$mine[$key][] = $second;
continue 3; // start streaming again with the new request in the set
}
}
continue;
}
if ($chunk->isLast()) {
// first successful answer for this key wins; cancel its sibling
}
}
}
That continue 3 is the bit I got wrong twice. You can’t add a response to a stream that’s already being iterated, so you have to break out and call stream() again with the bigger set.
The failure case you have to think about
If the second request fails fast, with a 500 say, it must not displace the original that’s still working. And the reverse: if the original dies while its twin is still running, you wait for the twin rather than starting a retry.
So when a response finishes, the question isn’t “did this one succeed?” It’s “does this key still have anything else running?” Only when the answer is no does a failure count as the key’s final answer. The test for this was the one I enjoyed writing most: a scripted client where one response stays silent for forty ticks and another fails immediately, to prove the slow one is still allowed to win.
Testing a clock you can’t wait for
You can’t unit test “100 seconds have passed” by waiting 100 seconds. The client takes a clock as a constructor argument, and the tests hand it one that jumps 30 seconds every time it’s read. Same trick for the sleep function used in the retry backoff. If a class reads the time directly, you can’t test what it does about the time.
There was a wrinkle. Symfony’s own mock responses turn an idle timeout into an error, so they can’t simulate “silent for a while, then answers”. I ended up writing a small scripted response class that stays quiet for N ticks. About forty lines, and worth it.
What it cost
Nothing visible on the days nothing stalls. On the days something does, a build that used to take seven minutes takes about three. A hedged call is flagged on the usage page as a retry, so I can see how often it happens, and so far the answer is “less often than I feared, more often than I’d like”.
A parallel job is as slow as its slowest member. If you can’t make the slow one faster, you can at least stop being loyal to it.