Streaming with FastAPI StreamingResponse
What You'll Learn
You'll learn how to implement streaming endpoints in FastAPI using StreamingResponse to progressively send AI-generated content to clients as it's produced.
The Core Idea
When an LLM generates text, it produces tokens sequentially rather than all at once. StreamingResponse lets you send these tokens to the client immediately as they arrive, creating a responsive user experience instead of making users wait for complete generation.
Think of it like a live news broadcast versus a recorded show. Instead of buffering the entire response and sending it in one chunk, you're broadcasting each piece as soon as it's ready.
How It Works
StreamingResponse accepts an async generator function that yields chunks of data. FastAPI automatically handles the chunked transfer encoding (which you learned about in the previous lesson) behind the scenes.
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import asyncio
app = FastAPI()
async def token_generator():
"""Simulates streaming tokens from an LLM"""
tokens = ["Hello", " world", "!", " How", " can", " I", " help?"]
for token in tokens:
yield token
await asyncio.sleep(0.1) # Simulate generation delay
@app.get("/stream")
async def stream_response():
return StreamingResponse(
token_generator(),
media_type="text/plain"
)
The key is your generator function: it yields data incrementally, and StreamingResponse sends each chunk immediately to the client. The client receives and can display each piece as it arrives—no waiting for completion.
For SSE format (from the previous lesson), you'd yield formatted SSE messages instead of plain text chunks.
Key Takeaway: StreamingResponse with async generators enables real-time streaming of LLM outputs, transforming batch responses into progressive, user-friendly experiences by sending tokens as they're generated.