Course contentsShow
← AI Engineering
Lesson 1618 of 1,88639. Classical ML Deployment (Non-LLM)Free lesson

Health Checks and Graceful Shutdown

Implement liveness and readiness probes, handle in-flight requests, and ensure zero-downtime deployments.

Health Checks and Graceful Shutdown

What you'll learn: How to keep your deployed ML models reliable and user-friendly by implementing proper health monitoring and zero-downtime deployment strategies.

Why This Matters

When you deploy a model serving API, the infrastructure needs to know when your service is ready to accept traffic and when it's still alive but temporarily unavailable. Without proper health checks and shutdown procedures, you risk dropping user requests during deployments or routing traffic to unhealthy instances.

Core Concepts

Liveness vs Readiness Probes

Think of liveness as "Is the process alive?" and readiness as "Is it ready to serve requests?"

  • Liveness probe: Checks if your service needs to be restarted (e.g., deadlock detection)
  • Readiness probe: Checks if your model is loaded and can handle requests (e.g., model warm-up complete)

A service can be alive but not ready—like a coffee shop that's open (alive) but still brewing the first pot (not ready).

Graceful Shutdown

When updating your model deployment, you need to:

  1. Stop accepting new requests (mark readiness as false)
  2. Wait for in-flight requests to complete (set a timeout, e.g., 30 seconds)
  3. Clean up resources (close database connections, flush logs)
  4. Exit cleanly

This ensures zero-downtime deployments when combined with rolling updates or blue-green strategies you learned earlier.

Handling In-Flight Requests

Track active request counts. When a shutdown signal (SIGTERM) arrives, the service enters "draining mode"—finishing current predictions but rejecting new ones. This prevents the "half-processed request" problem.

Key Takeaway: Implementing liveness and readiness probes with graceful shutdown handling ensures your model serving infrastructure can safely restart, upgrade, and scale without dropping user requests or causing downtime.