Health Checks and Readiness Probes
What You'll Learn
You'll learn how to configure endpoint monitoring to ensure new model instances are fully operational before they receive production traffic, preventing failed predictions during deployment.
The Core Problem
When you deploy a new model version (canary, blue-green, or rolling), the container or service may start before the model is actually ready to serve predictions. Loading large models into memory, initializing GPUs, warming up inference pipelines, and compiling optimizations can take seconds or even minutes. If traffic hits the endpoint during this startup phase, requests will fail or time out.
Health checks and readiness probes solve this by continuously testing whether your model service is alive and ready to handle requests.
Two Types of Probes
Health/Liveness Probe
Checks if the service is alive. If it fails repeatedly, the orchestration system (Kubernetes, ECS, etc.) restarts the container. Think of this as "Is the process running?"
Readiness Probe
Checks if the service is ready to accept traffic. Until this passes, the load balancer won't route requests to this instance. Think of this as "Has the model loaded and is it ready to predict?"
Implementation Pattern
Your model service should expose dedicated endpoints:
/health– Returns 200 if the process is alive/ready– Returns 200 only after the model is loaded, GPU initialized, and a test inference succeeds
The readiness probe often runs a lightweight dummy prediction to verify the full inference pipeline works before declaring the instance ready.
Key Takeaway: Health checks confirm your service is alive, while readiness probes ensure the model is fully operational before receiving production traffic—critical for safe deployments and preventing user-facing errors during instance startup.