Hedged Requests and Speculative Execution
What you'll learn: How to reduce tail latency by sending duplicate requests proactively, before the original request fails or times out.
The Core Idea
Sometimes a single request to a service takes unusually long—maybe that server is overloaded, experiencing a GC pause, or dealing with slow disk I/O. Instead of waiting for it to timeout or fail, hedged requests (also called speculative execution) send a second identical request to a different replica after a short delay (like the 95th percentile latency). Whichever response arrives first wins; the other is discarded.
This isn't a retry—you don't wait for failure. You're hedging your bet by running requests in parallel after detecting slowness.
Why It Works
Think of it like ordering food from two nearby restaurants when you're starving: if one is slow, the other might be faster. You cancel the slower order when the first arrives.
The technique targets tail latency—those rare but painful slow requests that hurt user experience. By sending a hedged request to another replica, you dramatically improve P99 latency without increasing median latency much (most requests finish before the hedge fires).
Key Considerations
- Timing matters: Trigger the hedge too early and you waste resources; too late and you've already suffered the delay
- Idempotency is critical: Since both requests might complete, your operation must be idempotent (remember idempotency keys from earlier lessons)
- Cost tradeoff: You're trading increased backend load (~5-10% more requests) for better user-facing latency
- Works with retries: Combines well with retry budgets and circuit breakers you've already learned
Key Takeaway: Hedged requests proactively send duplicate requests to backup replicas after a delay threshold, reducing tail latency by racing multiple attempts instead of waiting for timeouts—at the cost of modest additional load.