Course contentsShow
← AI Engineering
Lesson 109 of 1,8863. API-First AI EngineeringFree lesson

Implementing Streaming with OpenAI API

Set up streaming responses using OpenAI's stream parameter and handle delta tokens in real-time.

Implementing Streaming with OpenAI API

What You'll Learn

You'll learn how to enable real-time token streaming in OpenAI API calls and process the incremental response chunks as they arrive.

The Core Concept

When you set stream=True in an OpenAI API request, instead of waiting for the complete response, you receive delta tokens—small pieces of text—as the model generates them. This creates the familiar "ChatGPT typing effect" that users see.

Think of it like streaming a video versus downloading the entire file first. With streaming, you start seeing content immediately while the rest is still being generated.

How It Works

The OpenAI client returns an iterator when streaming is enabled. Each iteration gives you a chunk containing a delta—the newly generated piece of text. You handle these deltas one at a time:

from openai import OpenAI

client = OpenAI()

stream = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "Explain streaming"}],
    stream=True  # Enable streaming
)

for chunk in stream:
    delta = chunk.choices[0].delta
    if delta.content:
        print(delta.content, end="", flush=True)

Each chunk follows the Server-Sent Events pattern you learned earlier. The delta object contains only the new content since the last chunk—not the full response so far. You concatenate these deltas to build the complete message.

Why This Matters

Streaming drastically improves perceived latency. Users see progress within milliseconds instead of waiting seconds for a complete response, making your application feel more responsive even though total generation time remains similar.

Key Takeaway: Setting stream=True enables real-time token delivery via deltas; iterate through chunks and accumulate delta.content to display responses as they generate.