Course contentsShow
← Machine Learning and Deep Learning
Lesson 2726 of 3,53859. Distributed Training: Data ParallelismFree lesson

Gradient Accumulation in DDP

Implementing gradient accumulation to simulate larger batch sizes in distributed settings.

Gradient Accumulation in DDP

What you'll learn: How to simulate larger effective batch sizes in distributed training by accumulating gradients over multiple micro-batches before synchronizing across workers.

The Core Concept

In Distributed Data Parallel (DDP) training, each worker processes its local batch and synchronizes gradients after every backward pass. But what if you want an even larger effective batch size without running out of GPU memory? Gradient accumulation lets you split your desired batch into smaller "micro-batches," accumulate gradients locally across multiple forward-backward passes, and only then synchronize across workers.

This is especially powerful in distributed settings because:

  • You can achieve massive effective batch sizes (e.g., 4 GPUs × 8 micro-batches × 32 samples = 1024 effective batch)
  • Memory constraints become less limiting
  • You maintain the same communication efficiency by reducing synchronization frequency

How It Works in DDP

The key insight: disable gradient synchronization during accumulation steps, then re-enable it for the final micro-batch. In PyTorch DDP, gradients normally sync after every backward call. By wrapping accumulation steps with model.no_sync(), you prevent premature all-reduce operations:

accumulation_steps = 4

for i, (data, target) in enumerate(dataloader):
    # Sync only on the last micro-batch
    if (i + 1) % accumulation_steps == 0:
        # Normal backward with sync
        loss = criterion(model(data), target)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
    else:
        # Accumulate without syncing
        with model.no_sync():
            loss = criterion(model(data), target)
            loss.backward()  # gradients accumulate locally

Key Takeaway: Gradient accumulation in DDP uses no_sync() to prevent costly gradient synchronization until multiple micro-batches are processed, enabling memory-efficient simulation of enormous batch sizes across distributed workers.