Gradient Accumulation for Stability
What you'll learn: How to simulate larger batch sizes by accumulating gradients across multiple mini-batches to reduce variance and stabilize training.
The Core Problem
When training with small mini-batches, gradient estimates can be noisy and highly variable. Each mini-batch gives you a slightly different "direction" to update your weights. This variance can make training unstable, especially in deep networks or with limited GPU memory that forces you to use tiny batches.
How Gradient Accumulation Works
Instead of updating weights after every mini-batch, gradient accumulation lets you:
- Compute gradients on a mini-batch
- Add (accumulate) those gradients to previously computed ones
- Repeat for N mini-batches
- Update weights once using the accumulated gradients
- Reset and repeat
This simulates training with a batch size N times larger than what actually fits in memory.
The Stability Benefit
Think of it like polling: asking 10 people gives noisy opinions, but asking 100 people (10 groups of 10) gives more reliable results. By accumulating gradients across multiple small batches, you get a smoother, more stable gradient estimate that better represents the true direction toward the minimum.
This is especially valuable when:
- Memory constraints force small batch sizes
- Training becomes unstable with high gradient variance
- You want the generalization benefits of large batches without the memory cost
In Practice
Instead of .backward() followed immediately by optimizer.step(), you call .backward() multiple times to accumulate gradients, then step once every N iterations.
Key Takeaway: Gradient accumulation lets you simulate larger, more stable batches by summing gradients across multiple forward-backward passes before updating weights, reducing variance without increasing memory usage.