Course contentsShow
← Machine Learning and Deep Learning
Lesson 2985 of 3,53865. LLM Inference EnginesFree lesson

Dynamic Batch Size Management

Understand how batch sizes change dynamically each iteration as sequences finish and new ones join.

Dynamic Batch Size Management

What you'll learn: How serving systems automatically adjust the number of requests processed together as individual sequences complete and new requests arrive.

The Challenge of Fluid Batch Composition

In traditional batch processing, you fix your batch size upfront—say, 8 requests—and process them all together. But in continuous batching for LLM inference, something more complex happens: the batch size changes every single iteration.

Why? Because different sequences finish at different times. One prompt might generate a 10-token response while another needs 200 tokens. As sequences reach their end-of-sequence token or hit length limits, they exit the batch. Simultaneously, new incoming requests from the queue can join the active batch immediately—no need to wait for all current sequences to finish.

How It Works in Practice

Imagine you start iteration 1 with a batch of 6 sequences. During this iteration:

  • Sequence #3 generates its final token and completes → batch shrinks to 5
  • Two new requests arrive from the queue → batch grows to 7
  • Sequence #1 also finishes → down to 6 for the next iteration

The serving engine recalculates batch size between every forward pass, checking for:

  1. Exits: Which sequences just finished?
  2. Entries: What new requests can we admit without exceeding memory/compute limits?

This dynamic sizing maximizes GPU utilization—you're always processing as many sequences as resources allow, rather than waiting for synchronized batch boundaries.

Resource Constraints

The admission control system you learned earlier enforces limits: you can't grow the batch infinitely. Available KV cache memory, maximum batch size configurations, and throughput targets all constrain how many new sequences join each iteration.

Key Takeaway: Dynamic batch size management continuously adjusts the active batch size between iterations by removing completed sequences and admitting new requests, maximizing resource utilization without artificial synchronization barriers.