Course contentsShow
← Machine Learning and Deep Learning
Lesson 2784 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

Gradient Accumulation with Distributed Training

How gradient accumulation interacts with DDP and FSDP, including proper synchronization and scaling across multiple GPUs.

Gradient Accumulation with Distributed Training

What you'll learn: How gradient accumulation interacts with DDP and FSDP, requiring careful synchronization and gradient scaling to maintain training correctness across multiple GPUs.

The Core Challenge

When you combine gradient accumulation with distributed training (DDP or FSDP), you're essentially creating two levels of gradient aggregation: accumulation across microbatches within each worker, and synchronization across workers. Getting this interaction right is crucial for correctness.

How It Works with DDP

In standard DDP, gradients synchronize after every backward pass via all-reduce. With gradient accumulation:

  1. Disable synchronization during accumulation steps (only the local gradients accumulate)
  2. Enable synchronization only on the final accumulation step
  3. Scale gradients by 1 / (accumulation_steps × world_size) to account for the total effective batch size

Think of it like a relay race where each runner (GPU) completes multiple laps (microbatches) before passing the baton (synchronizing). You only communicate once per full cycle, not after every lap.

FSDP Considerations

FSDP adds complexity because parameters are sharded. During accumulation:

  • Gradients still accumulate locally on each rank's parameter shards
  • The final accumulation step triggers reduce-scatter to aggregate gradients across ranks
  • Each rank ends up with correctly averaged gradients for its parameter shard

Proper Scaling

The effective learning rate depends on: physical_batch_size × accumulation_steps × num_gpus. Your optimizer step must account for all gradient contributions. Without proper scaling, your model trains as if it saw more data than it actually did.

Key Takeaway: Gradient accumulation with distributed training requires disabling inter-GPU synchronization during accumulation steps and carefully scaling gradients by the total number of microbatches across all workers to maintain training equivalence with large-batch training.