Server-Sent Events (SSE) for Streaming
What you'll learn: How to stream LLM responses token-by-token to clients in real-time using Server-Sent Events.
Why SSE for LLM Streaming?
When an LLM generates a response, waiting for the entire completion before showing anything creates a poor user experience. Imagine watching a blank screen for 10 seconds versus seeing words appear as they're generated—like a person typing. SSE enables this real-time streaming effect.
Server-Sent Events is a standard protocol where the server pushes data to the client over a single HTTP connection. Unlike WebSockets (which are bidirectional), SSE is one-way: perfect for streaming LLM tokens from server to client.
How SSE Works for LLM Streaming
Think of SSE as a TV broadcast: the server (station) sends a continuous stream of data (program), and clients (viewers) tune in to receive it. The connection stays open until the stream completes.
When streaming LLM responses:
- Client makes a request with
Accept: text/event-stream - Server immediately responds with
Content-Type: text/event-stream - As the LLM generates each token, you send it as an SSE event
- Client receives and displays tokens progressively
- Server closes connection when generation completes
Each SSE message follows this format:
data: {"token": "Hello", "done": false}\n\n
data: {"token": " world", "done": false}\n\n
data: {"token": "!", "done": true}\n\n
The double newline (\n\n) signals the end of each event.
Key Characteristics
- Built on standard HTTP (no special protocols)
- Automatic reconnection if connection drops
- Works through most firewalls and proxies
- Simpler than WebSockets for one-way streaming
- Compatible with FastAPI's
StreamingResponse
Key Takeaway: SSE provides a simple, standard way to stream LLM tokens to clients in real-time, dramatically improving perceived responsiveness without complex bidirectional protocols.