Async Request Hedging

In this lesson, you’ll learn how to implement "Async Request Hedging" to drastically reduce tail latency by racing multiple redundant requests against each other.

The Problem: Tail Latency

In distributed systems, even if a service is performant 99% of the time, the "long tail" of latency (the P99 or P99.9) can be devastating. A single overloaded node or a network hiccup can cause a request to hang. Instead of waiting for a single, potentially slow request to finish, request hedging involves firing multiple identical requests concurrently and accepting the result from whichever one finishes first.

How it Works in Rust

In Rust’s async ecosystem, we leverage tokio::select! or the futures::future::select_ok combinator to race multiple futures. The moment one succeeds, we discard the others.

Consider this simplified pattern:

use futures::future::{select_ok, BoxFuture};

async fn hedged_request(endpoints: Vec<String>) -> Result<Response, Error> {
    let mut tasks = Vec::new();

    for url in endpoints {
        // Create multiple concurrent futures
        tasks.push(Box::pin(make_async_call(url)));
    }

    // select_ok returns the result of the first future that completes successfully
    let (result, _others) = select_ok(tasks).await?;
    
    Ok(result)
}

Strategic Considerations

While powerful, hedging consumes more resources. If you send three requests instead of one, you are effectively tripling your traffic to the downstream service. To use this effectively:

1. Thresholds: Don't hedge every request. Usually, you should send the first request, wait for a short duration (e.g., the P95 latency threshold), and then fire the second "hedged" request if the first hasn't returned. This balances latency gains against resource consumption.
2. Idempotency: Ensure the underlying operation is idempotent. If you are performing a POST or DELETE, receiving multiple requests might cause side effects unless the server handles deduplication.
3. Cancellation: Ensure that when one request wins, the others are promptly dropped. Because Rust futures are lazy and cancelable, dropping the remaining futures effectively instructs the executor to stop polling those operations, saving server resources.

Key Takeaway: Async request hedging minimizes tail latency by racing multiple concurrent requests and accepting the first success, provided the downstream operations are idempotent and resource usage is monitored.