Error Handling in Streaming Responses
What You'll Learn
You'll learn how to gracefully handle errors that occur mid-stream when streaming LLM responses, ensuring clients aren't left hanging with incomplete data.
The Challenge of Mid-Stream Failures
When streaming responses, you can't use traditional HTTP error codes — the response has already started with a 200 OK status. If an error occurs halfway through (rate limit, connection drop, model failure), you need an in-band strategy to communicate the problem.
Sentinel Values for Error Signaling
A sentinel value is a special marker in your stream that signals an error has occurred. Since you're streaming chunks of text or JSON, you embed error information directly in the stream:
async def stream_with_error_handling():
try:
async for chunk in llm_stream():
yield f"data: {json.dumps({'content': chunk})}\n\n"
except Exception as e:
# Send error sentinel
error_data = {
"error": True,
"type": "provider_error",
"message": str(e)
}
yield f"data: {json.dumps(error_data)}\n\n"
Connection Recovery Strategies
Your client should detect sentinels and decide whether to retry:
# Client-side handling
if chunk.get("error"):
if chunk["type"] == "rate_limit":
await asyncio.sleep(5) # Retry after backoff
return await retry_stream()
else:
raise StreamingError(chunk["message"])
Think of it like a radio broadcast — if transmission fails, you send an audible tone so listeners know to retune rather than assuming silence is content.
Key Takeaway: Handle streaming errors with sentinel values embedded in the stream itself, and implement client-side logic to detect errors and retry when appropriate.