Course contentsShow
← Machine Learning and Deep Learning
Lesson 2925 of 3,53863. Model Serving and Inference InfrastructureFree lesson

Latency vs Throughput: The Fundamental Tradeoff

Understanding the inverse relationship between latency and throughput, and when to optimize for each metric.

Latency vs Throughput: The Fundamental Tradeoff

You'll learn: Why optimizing for fast individual predictions often conflicts with maximizing total system capacity, and how to choose the right balance.

The Core Tension

Latency measures how long a single inference request takes from start to finish—the time one user waits for a prediction. Throughput measures how many requests your system handles per second—your total capacity.

These metrics have an inverse relationship: optimizations that reduce latency often decrease throughput, and vice versa.

Why They Conflict

Think of a restaurant: serving each customer immediately (low latency) means chefs prepare one meal at a time. But batch-cooking multiple orders together (high throughput) makes each individual customer wait longer.

In ML serving:

  • Low latency: Process requests individually, minimal batching, no queuing → fewer requests/second
  • High throughput: Use dynamic batching, larger batch sizes, accept queuing delays → slower individual responses

When you configured batch sizes and timeouts in earlier lessons, you were navigating this exact tradeoff.

When to Optimize for Each

Optimize for latency when:

  • Real-time user interactions (chatbots, recommendation systems)
  • Strict SLAs (e.g., "predictions under 100ms")
  • User experience depends on responsiveness

Optimize for throughput when:

  • Batch processing pipelines
  • Cost efficiency matters more than individual speed
  • Background tasks without user waiting
  • High-volume, non-interactive workloads

The Sweet Spot

Most production systems need balance. You might accept 50ms extra latency (from batching) to triple your throughput and cut infrastructure costs by 60%. This is why batch timeout configuration matters—it's your knob for adjusting this tradeoff.

Key Takeaway: Latency and throughput are inversely related; optimizing one typically sacrifices the other. Choose based on whether individual response time or total system capacity matters more for your use case.