Health Checks and Readiness Probes
What you'll learn: How to expose endpoints that verify your model server is alive, ready to handle requests, and that all dependencies are available—critical for container orchestration systems like Kubernetes.
Why Health Monitoring Matters
Once your model serving infrastructure is deployed (building on your REST/gRPC endpoints and error handling), orchestration systems need to know: Is this server actually working? Without health checks, a crashed server might still receive traffic, leading to timeouts and poor user experience.
Think of it like a restaurant's "Open" sign—you need one signal showing you're open for business (liveness) and another confirming the kitchen is ready to cook (readiness).
Types of Health Endpoints
Liveness Probe
Answers: "Is the server process running?" A simple endpoint like /health or /ping that returns HTTP 200. If this fails repeatedly, the orchestrator restarts the container.
@app.get("/health")
def health_check():
return {"status": "alive"}
Readiness Probe
Answers: "Can the server handle requests right now?" Checks that:
- The model is loaded in memory
- Database connections are active
- Feature store dependencies are reachable
- GPU memory is available
@app.get("/ready")
def readiness_check():
if model is None:
return {"status": "not ready"}, 503
if not feature_store.is_connected():
return {"status": "not ready"}, 503
return {"status": "ready"}
Orchestration Integration
Kubernetes and similar systems poll these endpoints. Failed liveness checks trigger restarts; failed readiness checks remove the pod from load balancer rotation until it recovers.
Key Takeaway: Health checks and readiness probes provide lightweight endpoints that let orchestration systems intelligently manage your model serving infrastructure, ensuring traffic only reaches healthy, ready-to-serve instances.