Timeout Management for Long-Running Requests
What You'll Learn
You'll learn how to set and enforce timeouts at multiple layers of your AI API to prevent hanging connections and cascade failures.
Why Timeouts Matter in AI APIs
LLM requests can take seconds or even minutes to complete. Without proper timeouts, a slow upstream service (the LLM provider, database, or vector store) can cause your entire API to hang, blocking threads and exhausting connection pools. Users wait indefinitely, and your system becomes unresponsive.
Think of timeouts like safety nets at different heights of a trapeze act. If a performer falls at any level, there's a net to catch them before they hit the ground.
Multi-Level Timeout Strategy
You need timeouts at three critical layers:
1. Server-Level Timeout
Your FastAPI or web server should have an overall request timeout (e.g., 60 seconds). This is the outer boundary—no request should take longer than this.
2. LLM Call Timeout
Each individual call to your LLM provider should have its own timeout (e.g., 30 seconds). This prevents a single slow model inference from blocking your entire request handler.
3. Database/Storage Timeout
Your database queries and vector searches should also timeout (e.g., 10 seconds). A slow database query shouldn't consume your entire LLM timeout budget.
The Cascade Effect
Set timeouts in decreasing order: server timeout > LLM timeout > database timeout. Each layer should have breathing room to handle the timeout from the layer below and return a graceful error to the user.
For example: 60s server → 30s LLM → 10s database. If the database times out at 10s, you still have 20s to retry or return an error before the LLM timeout triggers.
Key Takeaway: Always implement timeouts at the server, LLM call, and database levels with carefully chosen durations that allow each layer to fail gracefully without cascading the entire system into unresponsiveness.