Course contentsShow
← Machine Learning and Deep Learning
Lesson 2756 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

Pipeline Parallelism Fundamentals

Learn how pipeline parallelism splits model layers across devices and executes microbatches in pipeline fashion.

Pipeline Parallelism Fundamentals

What you'll learn: How to split a large neural network across multiple devices and keep them all busy by processing different microbatches simultaneously in a pipeline.

The Core Idea

When your model is too large to fit on a single GPU, pipeline parallelism divides the model's layers vertically across devices. Device 1 handles layers 1-10, Device 2 handles layers 11-20, and so on. Instead of processing one batch at a time (which would leave most devices idle), the input batch is split into smaller microbatches that flow through the pipeline like an assembly line.

Think of it like a car factory: Station 1 installs the engine, Station 2 adds the body, Station 3 paints it. While Station 3 paints Car A, Station 2 works on Car B's body, and Station 1 installs Car C's engine—all simultaneously.

How It Works

  1. Model Partitioning: Consecutive layers are assigned to different devices
  2. Microbatch Creation: Split your training batch into smaller chunks (e.g., batch size 128 → 8 microbatches of 16)
  3. Pipeline Execution:
    • Microbatch 1 goes to Device 1
    • When it finishes, it moves to Device 2 while Microbatch 2 starts on Device 1
    • This continues, filling the pipeline
  4. Gradient Accumulation: Gradients from all microbatches are accumulated before updating weights

The key challenge is the pipeline bubble—the idle time at the start (filling) and end (draining) when not all devices are working. Smaller microbatches reduce this bubble but increase communication overhead.

Key Takeaway: Pipeline parallelism splits models by layers across devices and processes multiple microbatches concurrently to maximize device utilization, trading perfect efficiency for the ability to train models larger than any single GPU's memory.