Health Checks and Readiness Probes
What you'll learn: How to implement health check endpoints that tell orchestration systems whether your LLM service is truly ready to handle requests.
Why Health Checks Matter
When you deploy an LLM API, containers may start before models finish loading into memory. If orchestration systems (like Kubernetes) send traffic immediately, users get errors. Health checks prevent this by distinguishing between "the container is running" and "the service can actually process requests."
Think of it like a restaurant: the lights might be on (container running), but if the kitchen hasn't prepped ingredients or heated ovens (model not loaded), you can't serve customers yet.
Two Types of Checks
Liveness probes answer: "Is the process alive?" They detect crashes and trigger restarts.
Readiness probes answer: "Can this instance handle traffic?" They wait for model loading, connection pooling, and warm-up to complete.
Implementation Pattern
Using FastAPI's dependency injection and background tasks (concepts you already know), create endpoints that check actual service state:
from fastapi import FastAPI, status
app = FastAPI()
# Track model loading state
service_ready = {"ready": False}
@app.on_event("startup")
async def load_model():
# Simulate model loading
# In reality: load LLM client, warm cache, etc.
service_ready["ready"] = True
@app.get("/health/live")
async def liveness():
return {"status": "alive"}
@app.get("/health/ready")
async def readiness():
if not service_ready["ready"]:
return JSONResponse(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
content={"status": "not_ready"}
)
return {"status": "ready"}
The readiness endpoint returns 503 until your model is loaded. Orchestration systems poll this before routing traffic.
Key Takeaway: Health checks prevent premature traffic routing by exposing model loading status through dedicated endpoints, ensuring your LLM service only receives requests when truly ready to respond.