Course contentsShow
← AI Engineering
Lesson 1023 of 1,88625. Model Serving and Inference OptimizationFree lesson

Batching with vLLM and TGI

Configure batching parameters in vLLM and Text Generation Inference for optimal continuous batching performance.

Batching with vLLM and TGI

What you'll learn: How to configure batching parameters in vLLM and Text Generation Inference to maximize throughput with continuous batching.

The Core Idea

Both vLLM and TGI implement continuous batching (also called "iteration-level batching"), which allows new requests to join a batch as soon as previous ones finish—no waiting for entire batches to complete. However, you control how aggressive this batching is through configuration parameters that balance throughput, latency, and memory usage.

vLLM Batching Configuration

In vLLM, key parameters include:

  • max_num_seqs: Maximum number of sequences processed simultaneously (batch size ceiling)
  • max_num_batched_tokens: Token budget per forward pass—prevents memory overflow
  • max_seq_len: Maximum sequence length allowed

Example configuration:

from vllm import LLM

llm = LLM(
    model="meta-llama/Llama-2-7b-hf",
    max_num_seqs=32,           # Up to 32 concurrent sequences
    max_num_batched_tokens=4096 # Process 4K tokens per iteration
)

TGI Batching Configuration

Text Generation Inference uses:

  • max_batch_total_tokens: Total token budget for the batch
  • max_waiting_tokens: Maximum tokens to accumulate before forcing a batch
  • max_batch_prefill_tokens: Budget specifically for prompt processing

Launch with Docker:

docker run --gpus all \
  -e MAX_BATCH_TOTAL_TOKENS=8192 \
  -e MAX_WAITING_TOKENS=20 \
  ghcr.io/huggingface/text-generation-inference:latest

Tuning Strategy

Think of it like airport security lines: max_num_seqs sets how many people can be screened simultaneously, while max_num_batched_tokens limits total luggage weight. You want queues full enough for efficiency, but not so packed that wait times explode.

Key Takeaway: Configure vLLM's max_num_seqs and max_num_batched_tokens or TGI's max_batch_total_tokens based on your GPU memory and latency requirements—larger batches increase throughput but may increase per-request latency.