Course contentsShow
← AI Engineering
Lesson 899 of 1,88622. Testing and CI/CD for AI SystemsFree lesson

Performance and Latency Testing

Measuring and asserting on response times, token usage, and throughput in end-to-end AI workflows.

Performance and Latency Testing

You'll learn how to systematically measure and enforce acceptable response times, token consumption, and throughput for your AI systems before they hit production.

What It Is

Performance and latency testing for AI systems means quantifying how fast your pipeline responds and how efficiently it uses resources. Unlike traditional software where response time is relatively predictable, AI systems involve variable factors: LLM processing time, token generation speed, retrieval overhead, and external API calls. You need to measure these holistically and set thresholds.

Think of it like a restaurant promising meals within 30 minutes. You wouldn't just time the cooking—you'd measure ordering, prep, cooking, and serving combined. Similarly, you measure your entire AI workflow from user input to final response.

What to Measure

Response Time: Total end-to-end latency from request to complete response. This includes retrieval, LLM processing, and post-processing.

Token Usage: Both input and output tokens consumed per request. This affects both cost and generation time.

Throughput: Requests handled per second or minute. Critical for understanding capacity limits.

Time-to-First-Token (TTFT): How quickly the first output token appears—crucial for streaming experiences.

Implementation Pattern

In your E2E tests, add timing assertions:

import time

def test_rag_performance():
    start = time.time()
    response = rag_pipeline.query("What is ML?")
    duration = time.time() - start
    
    assert duration < 3.0, f"Response too slow: {duration}s"
    assert response.token_count < 500, "Too many tokens used"

Run these tests regularly to catch performance regressions before deployment.

Key Takeaway: Performance testing for AI systems requires measuring end-to-end latency, token consumption, and throughput with automated assertions to prevent production slowdowns and cost overruns.