Latency and Throughput Metrics
What you'll learn: How to measure and balance retrieval speed and capacity when deploying RAG systems in production.
Understanding Speed vs. Volume
You've learned how to evaluate retrieval quality with metrics like NDCG and MRR. But in production, you also need to measure performance: how fast your system responds and how many requests it can handle simultaneously.
Latency is the time it takes to complete a single retrieval request—from query submission to receiving results. Think of it as "how long does one customer wait?" Measured in milliseconds (ms) or seconds.
Throughput is the number of retrieval requests your system can process per unit of time—typically queries per second (QPS). Think of it as "how many customers can we serve simultaneously?"
The Trade-off Triangle
You face a three-way tension:
- Retrieval Quality: High NDCG, comprehensive reranking, multiple retrieval strategies
- Latency: Fast response times (e.g., <200ms for interactive applications)
- Throughput: High concurrent capacity without degrading latency
Improving one often degrades another. Reranking with cross-encoders boosts quality but adds 50-200ms latency. Dense vector search with large indexes slows down under high load.
Why This Matters
A RAG system with 95% recall but 5-second latency will frustrate users. Conversely, blazing-fast retrieval that returns irrelevant documents wastes downstream LLM tokens and produces poor answers.
Production systems require monitoring p50, p95, and p99 latencies (the time within which 50%, 95%, and 99% of requests complete) alongside average throughput. A spike in p99 latency reveals that some users experience unacceptable delays even when average performance looks fine.
Key Takeaway: Production RAG systems must balance retrieval quality metrics with latency and throughput constraints—measuring response times at different percentiles helps identify performance bottlenecks before they impact user experience.