Request Timeouts and Cancellation
What You'll Learn
You'll learn how to protect your LLM API from hanging indefinitely when a generation takes too long, and how to gracefully cancel requests when users navigate away or no longer need the response.
Why This Matters
LLM requests can be unpredictable. A simple prompt might return in 2 seconds, while a complex one could take 30+ seconds—or hang entirely due to network issues or provider outages. Without proper timeout handling, your API could tie up workers indefinitely, creating a cascading failure that impacts all users.
Think of it like ordering at a restaurant: if the kitchen doesn't respond within a reasonable time, you need the ability to cancel your order and free up your table for other diners.
Core Concepts
Server-side timeouts prevent your API from waiting forever on the LLM provider. FastAPI's async handlers combined with asyncio.wait_for() let you wrap LLM calls with a maximum duration:
import asyncio
from fastapi import HTTPException
async def generate_with_timeout(prompt: str, timeout: float = 30.0):
try:
return await asyncio.wait_for(
llm_client.generate(prompt),
timeout=timeout
)
except asyncio.TimeoutError:
raise HTTPException(status_code=504, detail="LLM request timed out")
Client cancellation happens when users close their browser or navigate away. FastAPI automatically detects disconnected clients, and you can check request.is_disconnected() to abort expensive operations early, saving compute and API costs.
Set timeouts based on your SLA and user expectations—streaming responses might tolerate 60 seconds, while synchronous endpoints should fail faster (10-15 seconds) to prevent poor UX.
Key Takeaway: Always implement request timeouts to prevent resource exhaustion from slow LLM calls, and respect client cancellations to avoid wasting compute on responses no one will receive.