Async Circuit Breakers

In this lesson, you will learn how to implement a stateful, asynchronous circuit breaker to prevent cascading failures in your distributed systems.

The Concept of the Async Circuit Breaker

When an asynchronous service begins failing, repeatedly calling it is often counterproductive—it wastes system resources and worsens the latency for the downstream service. An Async Circuit Breaker acts as a state machine that sits between your request logic and the target service.

It tracks the health of your calls using three states:

1. Closed: Everything is fine. Requests flow through normally.
2. Open: The service is failing. Requests are rejected immediately (fail-fast) without attempting the async call.
3. Half-Open: The breaker periodically allows a limited number of "probe" requests to see if the service has recovered.

Implementation Pattern

To manage this in an async context, you must share the state safely across tasks, typically using an Arc<Mutex<State>> or an atomic structure if you want to avoid lock contention.

pub async fn call_with_breaker<F, Fut, T>(breaker: Arc<CircuitBreaker>, f: F) -> Result<T, Error>
where
    F: FnOnce() -> Fut,
    Fut: Future<Output = Result<T, Error>>,
{
    if !breaker.check_state().await {
        return Err(Error::CircuitOpen);
    }

    match f().await {
        Ok(val) => {
            breaker.record_success().await;
            Ok(val)
        }
        Err(e) => {
            breaker.record_failure().await;
            Err(e)
        }
    }
}

In the example above, the check_state method would determine if we are in the "Open" state. If so, it returns an error immediately. If the call succeeds, we notify the breaker to reset its failure counter. If it fails, the breaker increments a failure threshold; once that threshold is breached, the breaker transitions to the "Open" state.

Why it Matters for Async

Because async tasks are concurrent, your breaker must be thread-safe. By using tokio::sync::Mutex or parking_lot::Mutex (if critical sections are very short), you ensure that multiple concurrent tasks correctly influence the shared state of the circuit. This prevents "thundering herd" problems where multiple tasks detect a failure simultaneously and overwhelm a struggling service before the breaker has time to trip.

Key Takeaway: A stateful async circuit breaker protects system stability by providing a fail-fast mechanism that dynamically transitions between operational states based on real-time error telemetry.