Course contentsShow
← Machine Learning and Deep Learning
Lesson 2928 of 3,53863. Model Serving and Inference InfrastructureFree lesson

Batching for Throughput: Static vs Dynamic

How static batching maximizes throughput at the cost of latency, and when to use each batching strategy.

Batching for Throughput: Static vs Dynamic

You'll learn how static and dynamic batching strategies trade off latency for throughput differently, and when each makes sense.

The Core Tradeoff

Both static and dynamic batching increase throughput by processing multiple requests together, but they differ fundamentally in when they form batches and how they handle the latency-throughput balance.

Static Batching

Static batching groups a fixed number of requests before processing, regardless of wait time. It's like waiting for a bus to fill completely before departing—maximum efficiency per trip, but passengers who arrive early wait longer.

Characteristics:

  • Fixed batch size (e.g., always 32 requests)
  • Predictable throughput and resource utilization
  • Higher average latency due to mandatory wait times
  • Best for offline/batch workloads where latency isn't critical

When to use: High-volume scenarios where you're willing to accept higher tail latencies for maximum GPU/TPU utilization. Perfect for overnight batch scoring or periodic bulk inference.

Dynamic Batching

Dynamic batching (covered earlier) collects requests opportunistically within a time window. It's like a bus that leaves when full or after a maximum wait time—balancing passenger convenience with efficiency.

Characteristics:

  • Variable batch sizes (1 to max_batch_size)
  • Lower latency for early requests
  • Adapts to traffic patterns automatically
  • Slightly lower peak throughput than static

When to use: Online serving where user-facing latency matters. The timeout ensures no request waits indefinitely, while still capturing batching benefits during traffic spikes.

The Decision Matrix

Choose static batching when throughput maximization justifies adding seconds of latency. Choose dynamic batching when you need to balance both, with latency as a hard constraint.

Key Takeaway: Static batching maximizes throughput at the cost of predictable, often high latency by always waiting for full batches; use it for offline workloads. Dynamic batching adapts batch size to traffic, keeping latency bounded while capturing opportunistic throughput gains—ideal for online serving.