Course contentsShow
← System Design
Lesson 1339 of 1,91933. Reliability Engineering FoundationsFree lesson

Health Checks and Failure Detection

Implementing probes and heartbeats to quickly identify failed components and trigger failover.

Health Checks and Failure Detection

What You'll Learn

You'll learn how to implement probes and heartbeats to quickly detect when components fail and automatically trigger failover mechanisms.

The Core Idea

Imagine you're a manager checking on your team. You can't just assume everyone is working—you need regular check-ins. In distributed systems, health checks and heartbeats serve the same purpose: actively verifying that components are alive and functioning.

When you've built redundancy (N+1, active-passive, geographic), you still need a way to detect failures quickly. The faster you detect a failed component, the faster you can route traffic away from it and trigger failover to healthy replicas.

Two Main Approaches

Health Checks (Pull Model) A monitoring system or load balancer periodically queries each component: "Are you healthy?" The component responds with status information. If it fails to respond or returns an error, it's marked unhealthy.

Heartbeats (Push Model) Each component regularly sends "I'm alive" signals to a central coordinator. If heartbeats stop arriving, the component is presumed dead.

What Makes a Good Health Check?

  • Fast: Check frequently (every few seconds) to minimize detection time
  • Lightweight: Don't overload the system with expensive checks
  • Meaningful: Verify real functionality, not just "process is running"
  • Actionable: Should trigger automatic remediation (failover, alerts)

Most systems use shallow checks (basic liveness) for frequent monitoring and deep checks (database connectivity, dependencies) less frequently or on-demand.

Detection Time vs False Positives

There's a tradeoff: aggressive checks detect failures quickly but risk false alarms from transient network issues. Many systems require multiple consecutive failures before marking a component down.

Key Takeaway: Health checks and heartbeats are your early warning system—implement them thoughtfully to detect failures quickly while avoiding false positives that could trigger unnecessary failovers.