Course contentsShow
← AI Engineering
Lesson 971 of 1,88624. Building AI APIsFree lesson

Request Timeouts and Cancellation

Handle long-running LLM requests with proper timeout configuration and graceful cancellation support.

Request Timeouts and Cancellation

What You'll Learn

You'll learn how to protect your LLM API from hanging indefinitely when a generation takes too long, and how to gracefully cancel requests when users navigate away or no longer need the response.

Why This Matters

LLM requests can be unpredictable. A simple prompt might return in 2 seconds, while a complex one could take 30+ seconds—or hang entirely due to network issues or provider outages. Without proper timeout handling, your API could tie up workers indefinitely, creating a cascading failure that impacts all users.

Think of it like ordering at a restaurant: if the kitchen doesn't respond within a reasonable time, you need the ability to cancel your order and free up your table for other diners.

Core Concepts

Server-side timeouts prevent your API from waiting forever on the LLM provider. FastAPI's async handlers combined with asyncio.wait_for() let you wrap LLM calls with a maximum duration:

import asyncio
from fastapi import HTTPException

async def generate_with_timeout(prompt: str, timeout: float = 30.0):
    try:
        return await asyncio.wait_for(
            llm_client.generate(prompt),
            timeout=timeout
        )
    except asyncio.TimeoutError:
        raise HTTPException(status_code=504, detail="LLM request timed out")

Client cancellation happens when users close their browser or navigate away. FastAPI automatically detects disconnected clients, and you can check request.is_disconnected() to abort expensive operations early, saving compute and API costs.

Set timeouts based on your SLA and user expectations—streaming responses might tolerate 60 seconds, while synchronous endpoints should fail faster (10-15 seconds) to prevent poor UX.

Key Takeaway: Always implement request timeouts to prevent resource exhaustion from slow LLM calls, and respect client cancellations to avoid wasting compute on responses no one will receive.