In this lesson, you will learn how to configure circuit breakers within your service mesh to automatically trip and stop sending requests to failing services, preventing cascading system collapse.
The Concept of a Circuit Breaker
When a service begins to fail, sending it more traffic is often counterproductive. It keeps the service under high load, causing latency to spike and potentially crashing dependent services—a phenomenon known as a "cascading failure." A circuit breaker acts like an electrical fuse: when errors exceed a predefined threshold (such as a 50% failure rate over 10 seconds), the "circuit opens."
Once open, the mesh stops routing traffic to the unhealthy instance entirely, returning an immediate error to the caller. This provides the struggling service a "breathing room" to recover without being overwhelmed by incoming requests.
Configuring Circuit Breakers in Istio
In an Istio-enabled cluster, we use a DestinationRule to define these boundaries. By configuring an outlierDetection policy, we instruct the mesh to eject hosts from the load balancing pool if they return specific error codes.
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: order-service-breaker
spec:
host: order-service
trafficPolicy:
outlierDetection:
consecutive5xxErrors: 3
interval: 10s
baseEjectionTime: 30s
maxEjectionPercent: 50
In this example:
consecutive5xxErrors: 3: If a service instance fails three times in a row, it is flagged.interval: 10s: The mesh checks for failures every 10 seconds.baseEjectionTime: 30s: Once tripped, the instance is removed from the load balancer for 30 seconds.maxEjectionPercent: 50: Ensures that even if many instances are failing, at least half of the cluster remains available to handle traffic.
By offloading this logic to the service mesh sidecar, your application code stays clean of complex retry-logic or error-tracking, while your system gains an automated "fail-fast" mechanism that protects your infrastructure's health.
Key Takeaway: Circuit breakers prevent system-wide outages by proactively isolating failing services, allowing them to recover rather than exacerbating their instability with continued traffic.