Course contentsShow
← Machine Learning and Deep Learning
Lesson 2982 of 3,53865. LLM Inference EnginesFree lesson

The Iteration-Level Batching Problem

Learn why different sequences in a batch finish at different times, wasting GPU resources.

The Iteration-Level Batching Problem

What you'll learn: You'll understand why batching sequences together for LLM inference creates GPU inefficiency as sequences finish at different times.

The Core Challenge

When serving Large Language Models, we batch multiple requests together to maximize GPU throughput (as you learned in static vs dynamic batching). However, there's a hidden inefficiency: different sequences in the same batch complete at different token counts.

Imagine you're batching 8 user prompts together. One prompt might generate a 10-token response ("Yes, that's correct."), while another generates a 200-token essay. The problem? The entire batch must wait for the longest sequence to finish before releasing GPU resources.

Why This Wastes Resources

During autoregressive generation, the model processes one token at a time per sequence. Once a sequence emits its <EOS> (end-of-sequence) token, it's done—but it still occupies:

  • GPU memory for its KV cache
  • Batch slot position
  • Compute cycles (padding operations)

The GPU continues running attention and matrix multiplications for these "dead" sequences until the slowest sequence finishes. If you have 7 sequences done and 1 still generating, you're wasting ~87.5% of your compute on padding!

The Analogy

Think of a restaurant table seating a group. Even if 7 people finish their meals, the table (GPU batch slot) can't be used for new customers until the last person finishes dessert. Meanwhile, other hungry customers (waiting requests) stand idle outside.

Key Takeaway: Iteration-level batching inefficiency occurs because sequences in a batch complete at different times, forcing the GPU to waste resources on finished sequences while waiting for the longest one—a major motivation for more advanced batching strategies like continuous batching.