Health Checks and Readiness Probes
What You'll Learn
You'll learn how to implement liveness and readiness probes to ensure your containerized AI models are healthy and ready to serve traffic before Kubernetes routes requests to them.
Why This Matters in AI Deployments
In cloud environments, your model containers can fail in subtle ways: a model might be loading slowly, GPU memory could be exhausted, or the inference server might crash mid-request. Without proper health checks, Kubernetes might send traffic to broken containers, resulting in failed predictions and poor user experience.
Think of health checks like a doctor's examination: liveness probes check if the patient (container) is alive, while readiness probes verify they're actually ready to work. A model container might be alive but still loading a 70GB model file into memory—it's not ready for traffic yet.
The Two Types of Probes
Liveness Probes
Detect when a container is stuck or crashed. If a liveness probe fails, Kubernetes restarts the container. For AI models, this catches scenarios like:
- Inference server process crashed
- GPU driver hung
- Application deadlock
Readiness Probes
Determine when a container should receive traffic. If a readiness probe fails, Kubernetes temporarily removes the pod from service routing. Critical for:
- Model still loading from disk
- GPU initialization in progress
- Warming up KV cache or other initialization tasks
Common Probe Mechanisms
You can implement probes as:
- HTTP endpoints:
/healthreturns 200 when ready - TCP socket checks: Verify port is accepting connections
- Command execution: Run script inside container to verify model loaded
Key Takeaway: Health checks and readiness probes prevent user-facing errors by ensuring Kubernetes only routes traffic to fully initialized, functioning model containers—essential for reliable AI service deployments.