Course contentsShow
← AI Engineering
Lesson 1018 of 1,88625. Model Serving and Inference OptimizationFree lesson

Continuous Batching Fundamentals

Learn how continuous batching processes requests as they complete, maximizing GPU utilization without waiting for full batches.

Continuous Batching Fundamentals

What you'll learn: How continuous batching maximizes GPU utilization by processing requests as they complete, rather than waiting for entire batches to finish.

The Problem with Static Batching

In static batching (which you learned about in the previous lesson), the inference server waits until it collects a fixed number of requests before processing them together. The batch only completes when every request in it finishes generating tokens. This creates a problem: shorter requests sit idle waiting for the longest request to complete, wasting valuable GPU cycles.

How Continuous Batching Works

Continuous batching treats request completion dynamically. As soon as one request finishes generating its output, that slot immediately becomes available for a new incoming request—without waiting for the entire batch to complete.

Think of it like a ride-share van: instead of waiting until all passengers reach their destinations before picking up new riders, you drop off passengers as you reach their stops and immediately pick up new ones along the route. The van stays full and productive.

The Efficiency Gain

This approach dramatically improves GPU utilization because:

  • No GPU resources sit idle waiting for slower requests
  • New requests don't wait for arbitrary batch boundaries
  • The GPU processes the maximum number of requests it can handle at any moment
  • Average latency decreases since requests aren't blocked by unrelated slow completions

Modern LLM serving frameworks like vLLM and TGI (which you've already learned about) implement continuous batching by default, achieving 10-20x higher throughput compared to static batching for typical workloads.

Key Takeaway: Continuous batching eliminates idle GPU time by immediately backfilling completed request slots with new requests, maximizing throughput without sacrificing individual request latency.