Course contentsShow
← AI Engineering
Lesson 940 of 1,88623. LLM Application Architecture PatternsFree lesson

Timeout and Cancellation Handling

Managing long-running requests with timeouts, cancellation tokens, and graceful degradation strategies.

Timeout and Cancellation Handling

What you'll learn: How to protect your LLM application from hanging indefinitely by implementing timeouts, cancellation tokens, and graceful fallbacks when requests take too long.

The Problem with Long-Running Requests

LLM requests can be unpredictable. A simple chat completion might take 2 seconds—or 30. Without safeguards, your application can freeze waiting for responses that may never complete, leaving users frustrated and your servers locked up.

Core Timeout Strategies

Timeouts are your first line of defense. Set maximum wait times for API calls so requests fail fast rather than hang forever:

import asyncio

async def call_llm_with_timeout():
    try:
        response = await asyncio.wait_for(
            llm_api_call(),
            timeout=10.0  # 10-second limit
        )
        return response
    except asyncio.TimeoutError:
        return {"error": "Request timed out"}

Cancellation tokens let you abort operations mid-flight—think of them as an emergency stop button. When a user navigates away or a task becomes irrelevant, you can cancel the underlying work instead of wasting resources.

Graceful Degradation

When timeouts occur, don't just show an error. Implement graceful degradation:

  • Return cached results if available
  • Provide a simplified response from a faster model
  • Queue the request for background processing and notify the user later
  • Show partial results that arrived before the timeout

Analogy: Timeouts are like ordering food with a "30 minutes or free" policy. If your meal (LLM response) takes too long, you don't starve—you get your money back and order from somewhere faster.

Combining Strategies

Use timeouts with retry logic, background workers (from lesson 938), and async patterns (lesson 939) to build resilient systems that handle failures gracefully without blocking users.

Key Takeaway: Timeout and cancellation handling prevents your LLM application from hanging indefinitely by setting time limits, allowing manual cancellation, and providing fallback responses when operations exceed acceptable durations.