Testing Error Handling and Retries
What you'll learn: How to deliberately break your AI system's external dependencies to verify it fails gracefully and recovers intelligently.
Why This Matters
Your LLM application doesn't exist in a vacuum—it calls APIs that can fail, time out, or hit rate limits. Simply writing retry logic isn't enough; you must test that it actually works when things go wrong. Otherwise, your first production outage will reveal untested code paths.
The Core Concept
Testing error handling means simulating failures on purpose in your test environment. You'll use mocks, test doubles, or specialized testing tools to make your LLM API, embedding service, or vector database pretend to fail—then verify your code responds correctly.
Common failure scenarios to test:
- API errors (500, 503 status codes)
- Timeouts (request takes too long)
- Rate limits (429 status, quota exceeded)
- Transient network failures
- Partial failures (some tools work, others don't)
Example Test Structure
def test_llm_retry_on_rate_limit():
# Mock the LLM to fail twice with 429, then succeed
mock_llm = Mock()
mock_llm.generate.side_effect = [
RateLimitError("Too many requests"),
RateLimitError("Too many requests"),
"Success response"
]
# Your retry wrapper should handle this
result = call_with_retry(mock_llm, max_retries=3)
assert result == "Success response"
assert mock_llm.generate.call_count == 3
You're verifying the retry logic actually retries the right number of times and eventually succeeds (or fails gracefully after exhausting attempts).
Testing Timeout Behavior
Use mocks that sleep or stall to simulate slow responses, ensuring your timeouts trigger correctly and don't block indefinitely.
Key Takeaway: Deliberately simulate API failures, timeouts, and rate limits in tests to prove your error handling and retry logic work before production traffic hits them.