Course contentsShow
← Machine Learning and Deep Learning
Lesson 2983 of 3,53865. LLM Inference EnginesFree lesson

Continuous Batching Core Concept

Discover how continuous batching adds new requests to fill slots as old ones complete, maximizing throughput.

Continuous Batching Core Concept

What you'll learn: How continuous batching dynamically adds new requests to fill slots vacated by completed requests, maximizing GPU utilization and throughput.

The Core Problem with Iteration-Level Batching

In traditional iteration-level batching (which you learned about previously), all requests in a batch must complete together before a new batch can begin. This creates "bubbles" of idle GPU time when shorter sequences finish early but must wait for the longest sequence in the batch.

How Continuous Batching Works

Continuous batching treats the batch as a living, breathing queue rather than a fixed group. Here's the key insight:

  • Slot-based thinking: Instead of "batch 1, batch 2," think of the GPU as having slots (e.g., 32 request slots)
  • Dynamic replacement: When request #5 completes after 20 tokens, that slot immediately becomes available
  • Immediate backfilling: A new waiting request instantly fills the freed slot in the very next iteration
  • Per-request scheduling: Each request progresses at its own pace, generating tokens until completion

Think of it like a restaurant with continuous seating: as soon as one party leaves a table, the host seats the next waiting group. You don't wait until all tables are empty to seat new customers.

The Efficiency Gain

By eliminating the "wait for everyone" constraint, continuous batching achieves:

  • Higher GPU utilization: Fewer idle compute cycles
  • Better throughput: More requests processed per second
  • Lower queuing delays: Requests don't wait for entire batches to complete

The batch size becomes a fluid measure of how many requests are currently being processed, constantly changing as requests arrive and complete.

Key Takeaway: Continuous batching maximizes throughput by treating batching as a dynamic process where new requests immediately fill slots vacated by completed ones, eliminating the synchronization bottleneck of traditional static batching.