Course contentsShow
← Machine Learning and Deep Learning
Lesson 2757 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

GPipe: Microbatching and Pipeline Bubbles

Understand how microbatching reduces pipeline bubbles and improves GPU utilization in pipeline parallel training.

GPipe: Microbatching and Pipeline Bubbles

What You'll Learn

You'll understand how microbatching minimizes idle GPU time (pipeline bubbles) in pipeline parallel training, dramatically improving hardware utilization.

The Pipeline Bubble Problem

In pipeline parallelism, your model is split across multiple GPUs, with each handling specific layers. The challenge? When GPU-1 processes the forward pass and sends data to GPU-2, GPU-1 sits idle waiting for backward gradients to flow back—this idle time is a pipeline bubble.

Without optimization, the first GPU might be idle 75% of the time in a 4-stage pipeline!

Microbatching: The Solution

GPipe's key innovation is splitting each training batch into smaller microbatches. Instead of processing one large batch sequentially through the pipeline, you feed multiple microbatches in quick succession.

Analogy: Think of a car wash with four stations. With one car, three stations are always idle. But if you send four cars through in a row (microbatching), all stations stay busy most of the time.

How It Works

  1. Split your batch of size 128 into 8 microbatches of size 16
  2. GPU-1 processes microbatch 1, sends it to GPU-2
  3. Immediately GPU-1 starts on microbatch 2 (instead of waiting)
  4. GPU-2 processes microbatch 1 while GPU-1 handles microbatch 2
  5. Gradients accumulate across all microbatches before a weight update

The more microbatches you use, the smaller the bubble. GPipe achieves near-linear scaling by keeping all pipeline stages busy simultaneously.

The Tradeoff

More microbatches = better GPU utilization but slightly increased memory for storing intermediate activations. GPipe also uses activation checkpointing to manage this memory overhead.

Key Takeaway: GPipe uses microbatching to overlap computation across pipeline stages, dramatically reducing idle time (bubbles) and achieving high GPU utilization even with deep pipelines—essential for training models too large for single-device memory.