Course contentsShow
← AI Engineering
Lesson 981 of 1,88624. Building AI APIsFree lesson

Timeout Management for Long-Running Requests

Set and enforce request timeouts at multiple levels (server, LLM call, database) to prevent hanging connections.

Timeout Management for Long-Running Requests

What You'll Learn

You'll learn how to set and enforce timeouts at multiple layers of your AI API to prevent hanging connections and cascade failures.

Why Timeouts Matter in AI APIs

LLM requests can take seconds or even minutes to complete. Without proper timeouts, a slow upstream service (the LLM provider, database, or vector store) can cause your entire API to hang, blocking threads and exhausting connection pools. Users wait indefinitely, and your system becomes unresponsive.

Think of timeouts like safety nets at different heights of a trapeze act. If a performer falls at any level, there's a net to catch them before they hit the ground.

Multi-Level Timeout Strategy

You need timeouts at three critical layers:

1. Server-Level Timeout

Your FastAPI or web server should have an overall request timeout (e.g., 60 seconds). This is the outer boundary—no request should take longer than this.

2. LLM Call Timeout

Each individual call to your LLM provider should have its own timeout (e.g., 30 seconds). This prevents a single slow model inference from blocking your entire request handler.

3. Database/Storage Timeout

Your database queries and vector searches should also timeout (e.g., 10 seconds). A slow database query shouldn't consume your entire LLM timeout budget.

The Cascade Effect

Set timeouts in decreasing order: server timeout > LLM timeout > database timeout. Each layer should have breathing room to handle the timeout from the layer below and return a graceful error to the user.

For example: 60s server → 30s LLM → 10s database. If the database times out at 10s, you still have 20s to retry or return an error before the LLM timeout triggers.

Key Takeaway: Always implement timeouts at the server, LLM call, and database levels with carefully chosen durations that allow each layer to fail gracefully without cascading the entire system into unresponsiveness.