The Iteration-Level Batching Problem
What you'll learn: You'll understand why batching sequences together for LLM inference creates GPU inefficiency as sequences finish at different times.
The Core Challenge
When serving Large Language Models, we batch multiple requests together to maximize GPU throughput (as you learned in static vs dynamic batching). However, there's a hidden inefficiency: different sequences in the same batch complete at different token counts.
Imagine you're batching 8 user prompts together. One prompt might generate a 10-token response ("Yes, that's correct."), while another generates a 200-token essay. The problem? The entire batch must wait for the longest sequence to finish before releasing GPU resources.
Why This Wastes Resources
During autoregressive generation, the model processes one token at a time per sequence. Once a sequence emits its <EOS> (end-of-sequence) token, it's done—but it still occupies:
- GPU memory for its KV cache
- Batch slot position
- Compute cycles (padding operations)
The GPU continues running attention and matrix multiplications for these "dead" sequences until the slowest sequence finishes. If you have 7 sequences done and 1 still generating, you're wasting ~87.5% of your compute on padding!
The Analogy
Think of a restaurant table seating a group. Even if 7 people finish their meals, the table (GPU batch slot) can't be used for new customers until the last person finishes dessert. Meanwhile, other hungry customers (waiting requests) stand idle outside.
Key Takeaway: Iteration-level batching inefficiency occurs because sequences in a batch complete at different times, forcing the GPU to waste resources on finished sequences while waiting for the longest one—a major motivation for more advanced batching strategies like continuous batching.