Course contentsShow
← System Design
Lesson 1057 of 1,91927. Circuit Breakers and BulkheadsFree lesson

Failure Detection and Counting

Tracking failures with sliding windows, counting consecutive errors, and distinguishing retriable vs non-retriable failures.

Failure Detection and Counting

What you'll learn: How circuit breakers track and measure failures to make informed state-transition decisions.

The Core Challenge

A circuit breaker can't just flip states randomly—it needs reliable data. Failure detection and counting determines how the circuit breaker recognizes problems, accumulates evidence, and decides when a downstream service is unhealthy enough to open the circuit.

Sliding Windows vs Fixed Windows

You've seen these patterns before in rate limiting. Here they serve a different purpose:

  • Fixed window: Count failures within fixed time periods (e.g., "5 failures in the last 10 seconds"). Simple but can miss patterns at window boundaries.
  • Sliding window: Continuously tracks recent failures, providing smoother, more accurate detection as time progresses.

Sliding windows prevent the "edge effect" where a burst of errors spanning two fixed windows might not trigger the threshold in either window alone.

Consecutive vs Total Failures

Different counting strategies suit different scenarios:

  • Consecutive error count: "Open after 5 straight failures" — detects immediate, persistent issues.
  • Error rate: "Open when 50% of the last 20 requests fail" — better for intermittent problems mixed with successes.

Retriable vs Non-Retriable Failures

Not all failures should count the same:

  • Retriable failures: Timeouts, 503 Service Unavailable, network errors—suggest temporary downstream problems.
  • Non-retriable failures: 400 Bad Request, 401 Unauthorized—these are your fault, not the downstream service's. Don't count these toward opening the circuit.

This distinction prevents false positives where client bugs trigger circuit breaker trips.

Key Takeaway: Circuit breakers need intelligent failure detection—using sliding windows for accuracy, choosing appropriate counting strategies, and distinguishing between retriable system failures and non-retriable client errors ensures the circuit breaker opens only when it should.