Implementing Gradient Accumulation in PyTorch
You'll learn the exact mechanics of coding a gradient accumulation loop, including when to zero gradients and how to scale your loss.
The Core Implementation Pattern
Gradient accumulation splits one large batch into several smaller micro-batches, accumulating gradients across them before updating weights. The key is understanding three critical moments: when to zero gradients, when to scale loss, and when to step the optimizer.
Step-by-Step Loop Structure
Here's the canonical pattern:
optimizer.zero_grad() # Zero ONCE before accumulation starts
for i, (inputs, targets) in enumerate(dataloader):
outputs = model(inputs)
loss = criterion(outputs, targets)
# Scale loss by number of accumulation steps
loss = loss / accumulation_steps
loss.backward() # Accumulate gradients
# Update weights only every N steps
if (i + 1) % accumulation_steps == 0:
optimizer.step()
optimizer.zero_grad()
Critical Details
When to zero gradients: Only after optimizer steps, not after every backward pass. Otherwise you'd erase the accumulated gradients!
Why scale the loss: Without dividing by accumulation_steps, your effective learning rate would be multiplied by that factor. The scaling ensures gradients have the correct magnitude, as if you'd processed one large batch.
Think of it like filling a bucket: Each micro-batch adds water (gradients). You only pour it out (optimizer step) when the bucket is full, then empty it (zero_grad) for the next accumulation cycle.
Key Takeaway: Gradient accumulation requires zeroing gradients only after optimizer steps, scaling loss by the number of accumulation steps to maintain correct gradient magnitudes, and stepping the optimizer only when all micro-batches have contributed their gradients.