Implementing Streaming with OpenAI API
What You'll Learn
You'll learn how to enable real-time token streaming in OpenAI API calls and process the incremental response chunks as they arrive.
The Core Concept
When you set stream=True in an OpenAI API request, instead of waiting for the complete response, you receive delta tokens—small pieces of text—as the model generates them. This creates the familiar "ChatGPT typing effect" that users see.
Think of it like streaming a video versus downloading the entire file first. With streaming, you start seeing content immediately while the rest is still being generated.
How It Works
The OpenAI client returns an iterator when streaming is enabled. Each iteration gives you a chunk containing a delta—the newly generated piece of text. You handle these deltas one at a time:
from openai import OpenAI
client = OpenAI()
stream = client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "Explain streaming"}],
stream=True # Enable streaming
)
for chunk in stream:
delta = chunk.choices[0].delta
if delta.content:
print(delta.content, end="", flush=True)
Each chunk follows the Server-Sent Events pattern you learned earlier. The delta object contains only the new content since the last chunk—not the full response so far. You concatenate these deltas to build the complete message.
Why This Matters
Streaming drastically improves perceived latency. Users see progress within milliseconds instead of waiting seconds for a complete response, making your application feel more responsive even though total generation time remains similar.
Key Takeaway: Setting stream=True enables real-time token delivery via deltas; iterate through chunks and accumulate delta.content to display responses as they generate.