Course contentsShow
← Machine Learning and Deep Learning
Lesson 2764 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

Combining Pipeline and Tensor Parallelism

Understand 2D and 3D parallelism strategies that combine pipeline, tensor, and data parallelism simultaneously.

Combining Pipeline and Tensor Parallelism

What you'll learn: How to layer pipeline and tensor parallelism together to create 2D and 3D parallelism strategies that efficiently scale massive models across compute clusters.

The Core Idea

While pipeline parallelism splits a model vertically across layers and tensor parallelism splits individual layers horizontally across parameters, real-world distributed training often combines both. This creates 2D parallelism where each pipeline stage internally uses tensor parallelism, or 3D parallelism when you add data parallelism into the mix.

Think of it like organizing a factory: pipeline parallelism is the assembly line (different stations handle different stages), tensor parallelism is having multiple workers at each station sharing the work, and data parallelism is running multiple complete assembly lines in parallel.

Why Combine Them?

Pipeline parallelism alone creates idle GPU time (pipeline bubbles). Tensor parallelism alone requires heavy all-reduce communication within each layer. By combining them strategically:

  • Pipeline stages become smaller when layers are already split via tensor parallelism, reducing per-stage memory
  • Communication is localized within smaller GPU groups for tensor operations
  • Pipeline bubbles shrink with more flexible microbatch scheduling
  • Memory efficiency scales to models that fit neither approach alone

The Strategy

In 2D parallelism, you partition your GPU cluster into groups. Within each pipeline stage, a group of GPUs uses tensor parallelism (like Megatron-LM's column/row splitting). Across stages, pipeline parallelism (like GPipe or 1F1B) manages forward/backward scheduling. Each tensor parallel group acts as a single pipeline stage.

For 3D parallelism, add data parallelism on top: multiple independent 2D-parallelized model replicas process different data batches, synchronizing gradients at the end.

Key Takeaway: Combining pipeline and tensor parallelism creates multi-dimensional parallelism strategies that balance memory efficiency, communication overhead, and compute utilization for training models at unprecedented scale.