Gradient Accumulation in Pipeline Parallelism
You'll learn how gradients are accumulated across microbatches during pipeline parallel training to ensure correct optimization.
The Core Challenge
In pipeline parallelism, your model is split across multiple devices, with each processing a different stage. When you use microbatching (as in GPipe), each device processes multiple small batches before updating weights. But here's the problem: gradients from different microbatches must be accumulated before the optimizer step, just like in standard gradient accumulation—except now it happens across pipeline stages.
How It Works
Think of it like an assembly line where quality inspectors at each station collect feedback notes. Each worker (device) processes several items (microbatches) during their shift, writing down improvement suggestions (gradients) for each. Only when all items in the batch have passed through the entire pipeline do the workers combine their notes and make actual changes to their process.
The Mechanics
- Forward passes for all microbatches flow through the pipeline
- Backward passes return in reverse order
- Each stage accumulates gradients locally as its microbatches complete their backward pass
- Once all microbatches finish, gradients are synchronized across pipeline stages if needed
- A single optimizer step updates all parameters using the accumulated gradients
- Gradients are zeroed for the next batch
This accumulation is critical because each device only sees partial information. The gradient for a parameter reflects contributions from all microbatches that touched it, ensuring the optimization step is mathematically equivalent to processing one large batch.
Key Takeaway: Gradient accumulation in pipeline parallelism ensures that each device collects gradients from all its assigned microbatches before synchronizing and updating weights, maintaining training correctness despite the distributed, pipelined execution.