Course contentsShow
← Machine Learning and Deep Learning
Lesson 2912 of 3,53863. Model Serving and Inference InfrastructureFree lesson

Health Checks and Readiness Probes

Building robust health monitoring endpoints for Kubernetes deployments and load balancer integration.

Health Checks and Readiness Probes

What You'll Learn

You'll learn how to build robust health monitoring endpoints for Kubernetes deployments and load balancer integration in ML serving systems.

Why Health Endpoints Matter

When you deploy a model serving container, orchestrators like Kubernetes and load balancers need to know: "Is this container ready to handle requests?" and "Is it still healthy?" Without proper health checks, traffic might be routed to a container that's still loading a 10GB model file, or one that's crashed but hasn't been restarted yet.

Think of health checks like a restaurant's "Open" sign. Liveness probes check if your service is still alive (the restaurant exists). Readiness probes check if it's ready to serve customers (staff are present, kitchen is ready, model is loaded in memory).

Implementing Health Endpoints

Your serving API should expose two distinct endpoints:

Liveness endpoint (/health or /healthz): Returns 200 OK if the process is running. This tells Kubernetes whether to restart the container.

Readiness endpoint (/ready): Returns 200 OK only when your model is fully loaded, all dependencies are initialized, and the service can handle inference requests. During model loading, this should return 503.

# In your FastAPI serving application
@app.get("/health")
async def health():
    return {"status": "alive"}

@app.get("/ready")
async def ready():
    if not model_loaded or not preprocessor_ready:
        raise HTTPException(status_code=503, detail="Not ready")
    return {"status": "ready"}

These endpoints enable zero-downtime deployments: new pods only receive traffic after models load, and failing pods are automatically replaced.

Key Takeaway: Health checks and readiness probes are essential for production ML systems—they ensure load balancers and orchestrators only route traffic to containers that are truly ready to serve predictions, preventing failed requests during deployments or failures.