Course contentsShow
← AI Engineering
Lesson 888 of 1,88622. Testing and CI/CD for AI SystemsFree lesson

Testing Error Handling and Retries

Simulating API failures, timeouts, and rate limits to test resilience logic.

Testing Error Handling and Retries

What you'll learn: How to deliberately break your AI system's external dependencies to verify it fails gracefully and recovers intelligently.

Why This Matters

Your LLM application doesn't exist in a vacuum—it calls APIs that can fail, time out, or hit rate limits. Simply writing retry logic isn't enough; you must test that it actually works when things go wrong. Otherwise, your first production outage will reveal untested code paths.

The Core Concept

Testing error handling means simulating failures on purpose in your test environment. You'll use mocks, test doubles, or specialized testing tools to make your LLM API, embedding service, or vector database pretend to fail—then verify your code responds correctly.

Common failure scenarios to test:

  • API errors (500, 503 status codes)
  • Timeouts (request takes too long)
  • Rate limits (429 status, quota exceeded)
  • Transient network failures
  • Partial failures (some tools work, others don't)

Example Test Structure

def test_llm_retry_on_rate_limit():
    # Mock the LLM to fail twice with 429, then succeed
    mock_llm = Mock()
    mock_llm.generate.side_effect = [
        RateLimitError("Too many requests"),
        RateLimitError("Too many requests"),
        "Success response"
    ]
    
    # Your retry wrapper should handle this
    result = call_with_retry(mock_llm, max_retries=3)
    
    assert result == "Success response"
    assert mock_llm.generate.call_count == 3

You're verifying the retry logic actually retries the right number of times and eventually succeeds (or fails gracefully after exhausting attempts).

Testing Timeout Behavior

Use mocks that sleep or stall to simulate slow responses, ensuring your timeouts trigger correctly and don't block indefinitely.

Key Takeaway: Deliberately simulate API failures, timeouts, and rate limits in tests to prove your error handling and retry logic work before production traffic hits them.