Health Checks and Graceful Shutdown
What you'll learn: How to keep your deployed ML models reliable and user-friendly by implementing proper health monitoring and zero-downtime deployment strategies.
Why This Matters
When you deploy a model serving API, the infrastructure needs to know when your service is ready to accept traffic and when it's still alive but temporarily unavailable. Without proper health checks and shutdown procedures, you risk dropping user requests during deployments or routing traffic to unhealthy instances.
Core Concepts
Liveness vs Readiness Probes
Think of liveness as "Is the process alive?" and readiness as "Is it ready to serve requests?"
- Liveness probe: Checks if your service needs to be restarted (e.g., deadlock detection)
- Readiness probe: Checks if your model is loaded and can handle requests (e.g., model warm-up complete)
A service can be alive but not ready—like a coffee shop that's open (alive) but still brewing the first pot (not ready).
Graceful Shutdown
When updating your model deployment, you need to:
- Stop accepting new requests (mark readiness as false)
- Wait for in-flight requests to complete (set a timeout, e.g., 30 seconds)
- Clean up resources (close database connections, flush logs)
- Exit cleanly
This ensures zero-downtime deployments when combined with rolling updates or blue-green strategies you learned earlier.
Handling In-Flight Requests
Track active request counts. When a shutdown signal (SIGTERM) arrives, the service enters "draining mode"—finishing current predictions but rejecting new ones. This prevents the "half-processed request" problem.
Key Takeaway: Implementing liveness and readiness probes with graceful shutdown handling ensures your model serving infrastructure can safely restart, upgrade, and scale without dropping user requests or causing downtime.