Batching with vLLM and TGI
What you'll learn: How to configure batching parameters in vLLM and Text Generation Inference to maximize throughput with continuous batching.
The Core Idea
Both vLLM and TGI implement continuous batching (also called "iteration-level batching"), which allows new requests to join a batch as soon as previous ones finish—no waiting for entire batches to complete. However, you control how aggressive this batching is through configuration parameters that balance throughput, latency, and memory usage.
vLLM Batching Configuration
In vLLM, key parameters include:
max_num_seqs: Maximum number of sequences processed simultaneously (batch size ceiling)max_num_batched_tokens: Token budget per forward pass—prevents memory overflowmax_seq_len: Maximum sequence length allowed
Example configuration:
from vllm import LLM
llm = LLM(
model="meta-llama/Llama-2-7b-hf",
max_num_seqs=32, # Up to 32 concurrent sequences
max_num_batched_tokens=4096 # Process 4K tokens per iteration
)
TGI Batching Configuration
Text Generation Inference uses:
max_batch_total_tokens: Total token budget for the batchmax_waiting_tokens: Maximum tokens to accumulate before forcing a batchmax_batch_prefill_tokens: Budget specifically for prompt processing
Launch with Docker:
docker run --gpus all \
-e MAX_BATCH_TOTAL_TOKENS=8192 \
-e MAX_WAITING_TOKENS=20 \
ghcr.io/huggingface/text-generation-inference:latest
Tuning Strategy
Think of it like airport security lines: max_num_seqs sets how many people can be screened simultaneously, while max_num_batched_tokens limits total luggage weight. You want queues full enough for efficiency, but not so packed that wait times explode.
Key Takeaway: Configure vLLM's max_num_seqs and max_num_batched_tokens or TGI's max_batch_total_tokens based on your GPU memory and latency requirements—larger batches increase throughput but may increase per-request latency.