Course contentsShow
← AI Engineering
Lesson 967 of 1,88624. Building AI APIsFree lesson

Async Handlers for LLM Calls

Implement async endpoint handlers to avoid blocking the event loop during LLM inference calls.

Async Handlers for LLM Calls

What You'll Learn

You'll learn how to prevent your API from freezing while waiting for slow LLM responses by using asynchronous endpoint handlers.

The Problem with Synchronous Handlers

When you call an LLM service (like OpenAI), the request can take several seconds. If you use a regular synchronous handler, your FastAPI server sits idle, blocking the event loop. This means it can't handle any other incoming requests until the LLM responds—your API becomes a traffic jam where one slow car stops everyone behind it.

The Async Solution

By marking your endpoint handler as async and using await for LLM calls, you tell FastAPI: "While waiting for this LLM response, go handle other requests." The event loop stays free to process concurrent traffic.

The Pattern

from fastapi import FastAPI
from openai import AsyncOpenAI

app = FastAPI()
client = AsyncOpenAI()

@app.post("/chat")
async def chat_endpoint(message: str):
    # The 'await' releases control during the LLM call
    response = await client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": message}]
    )
    return {"reply": response.choices[0].message.content}

Notice the async def and await keywords. While this request waits for OpenAI, FastAPI can process dozens of other requests simultaneously.

Think of It Like a Restaurant

A synchronous handler is like a waiter who stands at the kitchen window staring until one order is ready. An async handler is like a waiter who takes multiple orders, checks the kitchen periodically, and serves whoever's food comes out first.

Key Takeaway: Always use async def handlers with await for LLM calls to keep your API responsive and maximize throughput under concurrent load.