1F1B Pipeline Schedule
What you'll learn: How the one-forward-one-backward (1F1B) schedule reduces memory overhead in pipeline parallelism by interleaving forward and backward passes more efficiently than GPipe's all-forward-then-all-backward approach.
The Memory Problem with GPipe
Recall that GPipe processes all microbatches in the forward pass before starting any backward passes. This means you must keep all intermediate activations for every microbatch in memory simultaneously until backward computation begins. For large models or many microbatches, this creates a massive memory burden.
How 1F1B Works
The one-forward-one-backward schedule solves this by alternating: after an initial warm-up phase, each pipeline stage performs one forward pass immediately followed by one backward pass for an earlier microbatch.
Think of it like a restaurant kitchen: instead of cooking all orders completely before plating any (GPipe), you start plating early orders while still cooking new ones (1F1B). This keeps your counter space (memory) from overflowing.
The Three Phases
- Warm-up: Initial forward passes fill the pipeline (no backward yet)
- Steady state: Alternate 1 forward + 1 backward (1F1B pattern)
- Cool-down: Final backward passes drain remaining activations
Memory Advantage
In steady state, you only store activations for one microbatch per stage instead of all microbatches. If you have 8 microbatches, 1F1B uses ~1/8th the activation memory of GPipe while maintaining the same pipeline efficiency.
The pipeline bubble (idle time) remains similar to GPipe, but memory efficiency improves dramatically—critical for scaling to very large models.
Key Takeaway: The 1F1B schedule interleaves forward and backward passes to free activation memory earlier, reducing peak memory usage by approximately the number of microbatches compared to GPipe's all-forward-then-all-backward approach.