Choosing Parallelism Dimensions
You'll learn: How to systematically decide on pipeline depth, tensor parallelism degree, and data parallelism size for optimal distributed training.
The Multi-Dimensional Decision
You now understand pipeline parallelism, tensor parallelism, data parallelism, ZeRO, and FSDP individually—but real deployments combine them. Choosing dimensions is like configuring a racing car: you have multiple knobs (pipeline stages, tensor parallel groups, data parallel workers), and turning one affects the others. The goal is maximizing throughput while fitting models in memory.
Core Heuristics
Start with memory constraints: Calculate your model's memory footprint. If one GPU can't hold parameters + optimizer states, you must use model parallelism (tensor or pipeline). Tensor parallelism splits individual layers—use it when layers themselves are huge (e.g., 175B+ parameter Transformers). Pipeline parallelism splits across layers—better for models with many moderate-sized layers.
Pipeline depth tradeoff: More stages = smaller per-GPU memory, but larger pipeline bubbles (idle time). Use 2–8 stages typically. Fewer, larger stages minimize bubbles; more stages help extremely large models.
Tensor parallelism degree: Powers of 2 (2, 4, 8) work best due to all-reduce efficiency. Higher degrees add communication overhead—keep within a single node when possible (NVLINK bandwidth helps). Use sparingly: 2–4x for most cases.
Data parallelism fills the rest: Once memory is satisfied via model parallelism, scale out remaining GPUs with data parallelism for throughput. This dimension has the best scaling efficiency since communication is less frequent.
Communication topology matters: Keep tensor parallelism within nodes (fast interconnect), pipeline parallelism across nodes (tolerates slower networking), data parallelism everywhere.
Key Takeaway:
Choose parallelism dimensions by first satisfying memory with minimal model parallelism, then scaling throughput with data parallelism—balancing memory footprint, communication overhead, and hardware topology.