Course contentsShow
← AI Engineering
Lesson 1110 of 1,88627. Cloud Deployment and ScalingFree lesson

Health Checks and Readiness Probes

Implementing liveness and readiness probes to ensure model containers are healthy and ready to serve traffic.

Health Checks and Readiness Probes

What You'll Learn

You'll learn how to implement liveness and readiness probes to ensure your containerized AI models are healthy and ready to serve traffic before Kubernetes routes requests to them.

Why This Matters in AI Deployments

In cloud environments, your model containers can fail in subtle ways: a model might be loading slowly, GPU memory could be exhausted, or the inference server might crash mid-request. Without proper health checks, Kubernetes might send traffic to broken containers, resulting in failed predictions and poor user experience.

Think of health checks like a doctor's examination: liveness probes check if the patient (container) is alive, while readiness probes verify they're actually ready to work. A model container might be alive but still loading a 70GB model file into memory—it's not ready for traffic yet.

The Two Types of Probes

Liveness Probes

Detect when a container is stuck or crashed. If a liveness probe fails, Kubernetes restarts the container. For AI models, this catches scenarios like:

  • Inference server process crashed
  • GPU driver hung
  • Application deadlock

Readiness Probes

Determine when a container should receive traffic. If a readiness probe fails, Kubernetes temporarily removes the pod from service routing. Critical for:

  • Model still loading from disk
  • GPU initialization in progress
  • Warming up KV cache or other initialization tasks

Common Probe Mechanisms

You can implement probes as:

  • HTTP endpoints: /health returns 200 when ready
  • TCP socket checks: Verify port is accepting connections
  • Command execution: Run script inside container to verify model loaded

Key Takeaway: Health checks and readiness probes prevent user-facing errors by ensuring Kubernetes only routes traffic to fully initialized, functioning model containers—essential for reliable AI service deployments.