Course contentsShow
← AI Engineering
Lesson 999 of 1,88624. Building AI APIsFree lesson

Error Handling in Streaming Responses

Manage errors mid-stream with sentinel values and connection recovery strategies.

Error Handling in Streaming Responses

What You'll Learn

You'll learn how to gracefully handle errors that occur mid-stream when streaming LLM responses, ensuring clients aren't left hanging with incomplete data.

The Challenge of Mid-Stream Failures

When streaming responses, you can't use traditional HTTP error codes — the response has already started with a 200 OK status. If an error occurs halfway through (rate limit, connection drop, model failure), you need an in-band strategy to communicate the problem.

Sentinel Values for Error Signaling

A sentinel value is a special marker in your stream that signals an error has occurred. Since you're streaming chunks of text or JSON, you embed error information directly in the stream:

async def stream_with_error_handling():
    try:
        async for chunk in llm_stream():
            yield f"data: {json.dumps({'content': chunk})}\n\n"
    except Exception as e:
        # Send error sentinel
        error_data = {
            "error": True,
            "type": "provider_error",
            "message": str(e)
        }
        yield f"data: {json.dumps(error_data)}\n\n"

Connection Recovery Strategies

Your client should detect sentinels and decide whether to retry:

# Client-side handling
if chunk.get("error"):
    if chunk["type"] == "rate_limit":
        await asyncio.sleep(5)  # Retry after backoff
        return await retry_stream()
    else:
        raise StreamingError(chunk["message"])

Think of it like a radio broadcast — if transmission fails, you send an audible tone so listeners know to retune rather than assuming silence is content.

Key Takeaway: Handle streaming errors with sentinel values embedded in the stream itself, and implement client-side logic to detect errors and retry when appropriate.