Course contentsShow
← Machine Learning and Deep Learning
Lesson 731 of 3,53817. Optimization for Deep LearningFree lesson

Gradient Accumulation for Stability

Using gradient accumulation to simulate larger batches and reduce gradient variance for stable training.

Gradient Accumulation for Stability

What you'll learn: How to simulate larger batch sizes by accumulating gradients across multiple mini-batches to reduce variance and stabilize training.

The Core Problem

When training with small mini-batches, gradient estimates can be noisy and highly variable. Each mini-batch gives you a slightly different "direction" to update your weights. This variance can make training unstable, especially in deep networks or with limited GPU memory that forces you to use tiny batches.

How Gradient Accumulation Works

Instead of updating weights after every mini-batch, gradient accumulation lets you:

  1. Compute gradients on a mini-batch
  2. Add (accumulate) those gradients to previously computed ones
  3. Repeat for N mini-batches
  4. Update weights once using the accumulated gradients
  5. Reset and repeat

This simulates training with a batch size N times larger than what actually fits in memory.

The Stability Benefit

Think of it like polling: asking 10 people gives noisy opinions, but asking 100 people (10 groups of 10) gives more reliable results. By accumulating gradients across multiple small batches, you get a smoother, more stable gradient estimate that better represents the true direction toward the minimum.

This is especially valuable when:

  • Memory constraints force small batch sizes
  • Training becomes unstable with high gradient variance
  • You want the generalization benefits of large batches without the memory cost

In Practice

Instead of .backward() followed immediately by optimizer.step(), you call .backward() multiple times to accumulate gradients, then step once every N iterations.

Key Takeaway: Gradient accumulation lets you simulate larger, more stable batches by summing gradients across multiple forward-backward passes before updating weights, reducing variance without increasing memory usage.