Course contentsShow
← Machine Learning and Deep Learning
Lesson 2927 of 3,53863. Model Serving and Inference InfrastructureFree lesson

Throughput Metrics and System Capacity

Measuring queries per second, token throughput, and understanding system saturation and capacity limits.

Throughput Metrics and System Capacity

What you'll learn: How to measure and understand your inference system's maximum processing capacity using throughput metrics.

Understanding Throughput

While latency measures how fast a single request completes, throughput measures how many requests your system can handle over time. It's the production line's speed, not just one item's journey time.

Think of throughput as your system's bandwidth: a highway might process thousands of cars per hour (high throughput) even if each car's individual trip takes 30 minutes (latency).

Key Throughput Metrics

Queries Per Second (QPS)

The most common throughput metric—how many inference requests your system processes per second. If your model serves 500 predictions in 10 seconds, your QPS is 50.

Token Throughput

For language models specifically, measuring tokens processed per second matters more than request count. A single request might generate 10 tokens or 1,000 tokens—vastly different computational loads.

System Saturation and Capacity Limits

As load increases, your system eventually hits saturation—the point where adding more requests doesn't increase throughput, only latency. This is your capacity limit.

Before saturation: More requests → more throughput
At saturation: More requests → same throughput, worse latency
Beyond saturation: System instability, timeouts, failures

Understanding capacity means load testing to find where your throughput plateaus. A system serving 100 QPS at 50ms latency might maintain that speed until 150 QPS, then latency explodes while throughput stalls—that's your ceiling.

Key Takeaway: Throughput metrics (QPS, tokens/second) reveal your system's processing capacity, while identifying saturation points helps you understand operational limits before performance degrades under load.