Course contentsShow
← Machine Learning and Deep Learning
Lesson 2901 of 3,53863. Model Serving and Inference InfrastructureFree lesson

Health Checks and Readiness Probes

Adding endpoints to verify server health, model readiness, and dependency availability for orchestration systems.

Health Checks and Readiness Probes

What you'll learn: How to expose endpoints that verify your model server is alive, ready to handle requests, and that all dependencies are available—critical for container orchestration systems like Kubernetes.

Why Health Monitoring Matters

Once your model serving infrastructure is deployed (building on your REST/gRPC endpoints and error handling), orchestration systems need to know: Is this server actually working? Without health checks, a crashed server might still receive traffic, leading to timeouts and poor user experience.

Think of it like a restaurant's "Open" sign—you need one signal showing you're open for business (liveness) and another confirming the kitchen is ready to cook (readiness).

Types of Health Endpoints

Liveness Probe

Answers: "Is the server process running?" A simple endpoint like /health or /ping that returns HTTP 200. If this fails repeatedly, the orchestrator restarts the container.

@app.get("/health")
def health_check():
    return {"status": "alive"}

Readiness Probe

Answers: "Can the server handle requests right now?" Checks that:

  • The model is loaded in memory
  • Database connections are active
  • Feature store dependencies are reachable
  • GPU memory is available
@app.get("/ready")
def readiness_check():
    if model is None:
        return {"status": "not ready"}, 503
    if not feature_store.is_connected():
        return {"status": "not ready"}, 503
    return {"status": "ready"}

Orchestration Integration

Kubernetes and similar systems poll these endpoints. Failed liveness checks trigger restarts; failed readiness checks remove the pod from load balancer rotation until it recovers.

Key Takeaway: Health checks and readiness probes provide lightweight endpoints that let orchestration systems intelligently manage your model serving infrastructure, ensuring traffic only reaches healthy, ready-to-serve instances.