What is Gradient Accumulation and Why It's Needed
You'll learn how gradient accumulation lets you simulate larger batch sizes when GPU memory is too small to fit your desired batch.
The Core Problem
When training large models, you often want to use large batch sizes for stable gradients and better convergence. However, GPU memory is limited—you might only fit 4 examples per batch when you really want 64. Loading all 64 at once would cause an out-of-memory error.
How Gradient Accumulation Works
Instead of computing gradients on all 64 examples simultaneously, gradient accumulation breaks them into smaller mini-batches (say, 4 examples each). Here's the key insight:
- Forward pass on mini-batch 1 → compute loss
- Backward pass → compute gradients, but don't update weights yet
- Accumulate (add) these gradients to a running total
- Repeat steps 1-3 for mini-batches 2, 3, ..., 16
- After all mini-batches, average the accumulated gradients
- Update model weights once with the averaged gradient
The Analogy
Think of it like taking multiple measurements to reduce noise. Rather than measuring once with a big ruler you can't fit, you measure 16 times with a small ruler and average the results—mathematically equivalent, but memory-friendly.
Why It's Critical
- Memory efficiency: Train with effective batch size of 64 while only using memory for 4
- Gradient stability: Larger effective batches mean less noisy gradient estimates
- Distributed training: Essential when combined with techniques like DDP or FSDP to maximize throughput without exceeding memory
The tradeoff? More computation time per weight update, since you're doing multiple forward/backward passes.
Key Takeaway: Gradient accumulation simulates large-batch training by summing gradients across multiple small mini-batches before updating weights, enabling memory-constrained training of large models.