Handling Partial Responses and Deltas
What you'll learn: How to process incomplete chunks of text as they arrive during streaming and build complete responses from fragments.
When you enable streaming with an LLM API (using Server-Sent Events), the model doesn't send you one complete response—it sends many tiny pieces called deltas. Each delta contains just a few tokens, sometimes even single characters. Your job is to catch these fragments, store them, and reassemble them into the full answer.
The Challenge of Incremental Updates
Think of streaming responses like receiving a letter one word at a time through a mail slot. Each envelope (delta) contains just a word or two. You need to:
- Open each envelope as it arrives (parse the delta)
- Read what's inside (extract the new token/text)
- Append it to your growing letter (accumulate state)
- Keep track of where you are (maintain position and context)
Maintaining State During Streaming
Unlike batch responses where you get everything at once, streaming requires you to maintain state—a variable that holds all accumulated text so far. Each time a new delta arrives, you add its content to your running total.
The delta typically contains only the new content, not everything repeated. For example:
- Delta 1: "The"
- Delta 2: " weather"
- Delta 3: " is"
- Accumulated result: "The weather is"
You must handle the case where deltas arrive out of order, connection drops mid-stream, or the final delta signals completion.
Key Takeaway: Streaming responses require you to accumulate partial deltas into complete text by maintaining state across multiple incremental updates, not just waiting for one finished response.