Speculative Execution and Racing Requests
What you'll learn: How to reduce tail latency by racing multiple requests simultaneously and using whichever finishes first.
The Core Idea
Speculative execution means sending the same request to multiple replicas or servers at the same time, then using the first successful response and canceling the others. This is an extension of hedged requests (which you've already learned) taken to its logical extreme—instead of waiting before hedging, you race all requests from the start.
Think of it like ordering a taxi from multiple ride-share apps simultaneously. The moment one driver arrives, you cancel the others. You pay a small cost (sending multiple requests) to guarantee the fastest possible response.
Why Race Requests?
Even with adaptive timeouts, a single slow replica can ruin user experience. If Service A has 3 replicas with P50 latency of 10ms but P99 of 500ms, any single request has a 1% chance of being terrible. By racing requests to all 3 replicas, you dramatically reduce the probability that all of them are slow simultaneously.
The Tradeoff
Racing requests increases load on your backend—you're doing 2x or 3x the work. This only makes sense when:
- The operation is read-only or idempotent
- Tail latency directly impacts user experience (critical path)
- You have spare backend capacity
- The latency improvement justifies the cost
You typically limit racing to the most critical, latency-sensitive operations rather than applying it everywhere.
Key Takeaway: Speculative execution races identical requests to multiple replicas simultaneously, using the fastest response to eliminate tail latency at the cost of increased backend load—reserve it for your most critical, latency-sensitive operations.