Probe Anti-Patterns and Common Mistakes
What you'll learn: You'll discover the common pitfalls that can make your Kubernetes health checks unreliable or even dangerous to cluster stability.
Even well-intentioned probe configurations can backfire spectacularly. Understanding what not to do is just as critical as knowing proper probe design.
Cascading Failures
The most dangerous anti-pattern is creating probes that depend on external services. If your liveness probe checks a database connection and the database becomes temporarily unavailable, all your pods restart simultaneously. This creates a thundering herd that can overwhelm the database when it recovers, preventing actual recovery. Think of it like everyone in a theater rushing the exit at once—nobody gets out efficiently.
Solution: Health checks should verify the pod's own ability to serve traffic, not external dependencies.
Expensive Health Checks
Probes that perform heavy computation, complex database queries, or call multiple downstream services waste resources and slow response times. Remember that probes run continuously (every few seconds) across potentially hundreds of pods.
Solution: Keep checks lightweight—simple HTTP endpoints that return quickly, ideally under 100ms.
Incorrect Thresholds
Setting failureThreshold: 1 with a short periodSeconds makes probes too sensitive to transient issues. Conversely, high thresholds delay detection of real problems. Similarly, insufficient initialDelaySeconds causes startup probes to kill slow-starting containers prematurely.
Solution: Tune thresholds based on actual application behavior, not guesses. Use startup probes for slow containers instead of inflating liveness delays.
Identical Liveness and Readiness Checks
Using the same endpoint for both probe types means a pod gets killed (liveness) when it should merely stop receiving traffic (readiness). These serve different purposes and should use different logic.
Key Takeaway: Avoid cascading failures by keeping probes independent of external dependencies, design lightweight checks that run efficiently at scale, and tune thresholds realistically to balance sensitivity with stability.