Course contentsShow
← Machine Learning and Deep Learning
Lesson 2759 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

1F1B Pipeline Schedule

Study the one-forward-one-backward schedule that improves memory efficiency over naive pipeline parallelism.

1F1B Pipeline Schedule

What you'll learn: How the one-forward-one-backward (1F1B) schedule reduces memory overhead in pipeline parallelism by interleaving forward and backward passes more efficiently than GPipe's all-forward-then-all-backward approach.

The Memory Problem with GPipe

Recall that GPipe processes all microbatches in the forward pass before starting any backward passes. This means you must keep all intermediate activations for every microbatch in memory simultaneously until backward computation begins. For large models or many microbatches, this creates a massive memory burden.

How 1F1B Works

The one-forward-one-backward schedule solves this by alternating: after an initial warm-up phase, each pipeline stage performs one forward pass immediately followed by one backward pass for an earlier microbatch.

Think of it like a restaurant kitchen: instead of cooking all orders completely before plating any (GPipe), you start plating early orders while still cooking new ones (1F1B). This keeps your counter space (memory) from overflowing.

The Three Phases

  1. Warm-up: Initial forward passes fill the pipeline (no backward yet)
  2. Steady state: Alternate 1 forward + 1 backward (1F1B pattern)
  3. Cool-down: Final backward passes drain remaining activations

Memory Advantage

In steady state, you only store activations for one microbatch per stage instead of all microbatches. If you have 8 microbatches, 1F1B uses ~1/8th the activation memory of GPipe while maintaining the same pipeline efficiency.

The pipeline bubble (idle time) remains similar to GPipe, but memory efficiency improves dramatically—critical for scaling to very large models.

Key Takeaway: The 1F1B schedule interleaves forward and backward passes to free activation memory earlier, reducing peak memory usage by approximately the number of microbatches compared to GPipe's all-forward-then-all-backward approach.