Testing FastAPI LLM Endpoints
What you'll learn: How to write robust tests for your FastAPI LLM endpoints without making expensive or slow real API calls.
Why Testing Matters for LLM APIs
Testing your LLM endpoints ensures they handle requests correctly, validate input properly, and return expected response structures—all without burning through API credits or waiting for slow inference during development.
Using FastAPI's TestClient
FastAPI provides TestClient, which lets you make requests to your API as if you were a real client, but entirely in-memory. Combined with mocks for LLM calls (you learned this earlier), you can validate endpoint behavior predictably:
from fastapi.testclient import TestClient
from unittest.mock import patch
from myapp import app # Your FastAPI app
client = TestClient(app)
def test_completion_endpoint():
# Mock the LLM response
with patch('myapp.openai_client.chat.completions.create') as mock_llm:
mock_llm.return_value.choices[0].message.content = "Mocked response"
response = client.post(
"/completions",
json={"prompt": "Hello", "max_tokens": 100}
)
assert response.status_code == 200
assert response.json()["text"] == "Mocked response"
mock_llm.assert_called_once()
What to Test
- Request validation: Send invalid Pydantic models and verify 422 errors
- Response structure: Ensure your response model serializes correctly
- Authentication: Test protected endpoints with valid/invalid credentials
- Error handling: Mock LLM failures and verify graceful degradation
- Background tasks: Verify logging tasks are queued (without executing them)
The Testing Advantage
By mocking LLM responses, your tests run in milliseconds, are deterministic, and cost nothing. You're testing your code—routing, validation, error handling—not the LLM provider's infrastructure.
Key Takeaway: Use TestClient with mocked LLM responses to write fast, reliable, cost-free tests that validate your FastAPI endpoint logic without real inference calls.