Gradient Accumulation in DDP
What you'll learn: How to simulate larger effective batch sizes in distributed training by accumulating gradients over multiple micro-batches before synchronizing across workers.
The Core Concept
In Distributed Data Parallel (DDP) training, each worker processes its local batch and synchronizes gradients after every backward pass. But what if you want an even larger effective batch size without running out of GPU memory? Gradient accumulation lets you split your desired batch into smaller "micro-batches," accumulate gradients locally across multiple forward-backward passes, and only then synchronize across workers.
This is especially powerful in distributed settings because:
- You can achieve massive effective batch sizes (e.g., 4 GPUs × 8 micro-batches × 32 samples = 1024 effective batch)
- Memory constraints become less limiting
- You maintain the same communication efficiency by reducing synchronization frequency
How It Works in DDP
The key insight: disable gradient synchronization during accumulation steps, then re-enable it for the final micro-batch. In PyTorch DDP, gradients normally sync after every backward call. By wrapping accumulation steps with model.no_sync(), you prevent premature all-reduce operations:
accumulation_steps = 4
for i, (data, target) in enumerate(dataloader):
# Sync only on the last micro-batch
if (i + 1) % accumulation_steps == 0:
# Normal backward with sync
loss = criterion(model(data), target)
loss.backward()
optimizer.step()
optimizer.zero_grad()
else:
# Accumulate without syncing
with model.no_sync():
loss = criterion(model(data), target)
loss.backward() # gradients accumulate locally
Key Takeaway: Gradient accumulation in DDP uses no_sync() to prevent costly gradient synchronization until multiple micro-batches are processed, enabling memory-efficient simulation of enormous batch sizes across distributed workers.