Course contentsShow
← System Design
Lesson 106 of 1,9193. Load BalancingFree lesson

Health Check False Positives and Flapping

Network blips or brief overloads can cause false failures; requiring multiple consecutive failures prevents unstable server rotation.

Health Check False Positives and Flapping

You'll learn how temporary network issues can cause unnecessary server removals and how requiring multiple failures prevents chaos in your load balancer.

The Problem: Unstable Server Rotation

Imagine a perfectly healthy server experiences a single momentary network hiccup—maybe a packet gets dropped or the server is briefly overloaded handling a spike. If your load balancer immediately marks it unhealthy after one failed health check, it stops routing traffic there. Then the health check succeeds again, traffic returns, another blip occurs, and the cycle repeats. This rapid on-off-on-off behavior is called flapping, and it creates instability across your entire system.

A false positive means the health check incorrectly reports a server as unhealthy when it's actually fine—just experiencing a temporary issue.

The Solution: Consecutive Failure Threshold

Instead of reacting to a single failure, load balancers typically require multiple consecutive failed health checks before marking a server unhealthy. For example, you might configure "3 consecutive failures over 30 seconds" as your threshold.

This is like a teacher not sending a student to the principal after one wrong answer—they wait to see if there's a pattern before taking action.

Similarly, you can require multiple consecutive successful checks before marking a previously unhealthy server as healthy again. This prevents premature re-entry.

Configuration Example

  • Failure threshold: 3 consecutive failures → mark unhealthy
  • Success threshold: 2 consecutive successes → mark healthy
  • Check interval: 10 seconds

This means a server needs 30 seconds of consistent failure signals before removal, filtering out temporary blips while still detecting real problems quickly enough.

Key Takeaway: Requiring multiple consecutive health check failures (and successes) prevents false positives and flapping, keeping your load balancer's server pool stable despite temporary network issues or brief overloads.