Designing for Failure with Retry and Circuit Breaker

Implement resilience patterns to prevent cascading failures in microservice environments.

In this lesson, you will learn how to prevent cascading failures in your microservices by implementing strategic retry policies and circuit breakers.

The Problem: Cascading Failure

In a distributed system, when one service becomes unresponsive, calling it repeatedly can lead to thread exhaustion and memory pressure, eventually causing your entire application to crash. To prevent this "thundering herd" effect, we use two complementary patterns: Retries and Circuit Breakers.

Implementing Resilience

A Retry policy helps handle transient faults (like temporary network blips). However, retrying blindly is dangerous—if the downstream service is actually down, you are simply hammering it while it tries to recover.

A Circuit Breaker acts as a safety switch. It monitors the failure rate of a service. If the failure rate crosses a specific threshold, the circuit "opens," and subsequent calls fail fast immediately without even attempting to reach the downstream service. This gives the failing service space to recover.

Here is how you might implement this logic using a standard policy approach in C#:

// Define a retry policy: retry 3 times with exponential backoff
var retryPolicy = HttpPolicyExtensions
    .HandleTransientHttpError()
    .WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));

// Define a circuit breaker: break the circuit if 50% of requests fail 
// within 30 seconds, and keep it open for 30 seconds
var circuitBreaker = HttpPolicyExtensions
    .HandleTransientHttpError()
    .AdvancedCircuitBreakerAsync(0.5, TimeSpan.FromSeconds(30), 10, TimeSpan.FromSeconds(30));

// Combine policies to wrap your HTTP calls
var resilienceStrategy = Policy.WrapAsync(retryPolicy, circuitBreaker);

Strategic Application

When designing for failure, always apply these patterns at the edge of your service boundaries. By wrapping your outbound HTTP calls, you ensure that your service remains responsive even when its dependencies are struggling. Remember: if the circuit is open, return a fallback response (such as cached data or a default value) rather than throwing an exception to the user. This "graceful degradation" is the hallmark of a robust microservice architecture.

Key Takeaway: Retries manage transient instability, while circuit breakers provide system-wide immunity to cascading failures by failing fast during prolonged outages.