Timeout and Cancellation Handling
What you'll learn: How to protect your LLM application from hanging indefinitely by implementing timeouts, cancellation tokens, and graceful fallbacks when requests take too long.
The Problem with Long-Running Requests
LLM requests can be unpredictable. A simple chat completion might take 2 seconds—or 30. Without safeguards, your application can freeze waiting for responses that may never complete, leaving users frustrated and your servers locked up.
Core Timeout Strategies
Timeouts are your first line of defense. Set maximum wait times for API calls so requests fail fast rather than hang forever:
import asyncio
async def call_llm_with_timeout():
try:
response = await asyncio.wait_for(
llm_api_call(),
timeout=10.0 # 10-second limit
)
return response
except asyncio.TimeoutError:
return {"error": "Request timed out"}
Cancellation tokens let you abort operations mid-flight—think of them as an emergency stop button. When a user navigates away or a task becomes irrelevant, you can cancel the underlying work instead of wasting resources.
Graceful Degradation
When timeouts occur, don't just show an error. Implement graceful degradation:
- Return cached results if available
- Provide a simplified response from a faster model
- Queue the request for background processing and notify the user later
- Show partial results that arrived before the timeout
Analogy: Timeouts are like ordering food with a "30 minutes or free" policy. If your meal (LLM response) takes too long, you don't starve—you get your money back and order from somewhere faster.
Combining Strategies
Use timeouts with retry logic, background workers (from lesson 938), and async patterns (lesson 939) to build resilient systems that handle failures gracefully without blocking users.
Key Takeaway: Timeout and cancellation handling prevents your LLM application from hanging indefinitely by setting time limits, allowing manual cancellation, and providing fallback responses when operations exceed acceptable durations.