Course contentsShow
← Machine Learning and Deep Learning
Lesson 2781 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

What is Gradient Accumulation and Why It's Needed

Understanding how gradient accumulation simulates larger batch sizes when GPU memory is limited by accumulating gradients over multiple mini-batches.

What is Gradient Accumulation and Why It's Needed

You'll learn how gradient accumulation lets you simulate larger batch sizes when GPU memory is too small to fit your desired batch.

The Core Problem

When training large models, you often want to use large batch sizes for stable gradients and better convergence. However, GPU memory is limited—you might only fit 4 examples per batch when you really want 64. Loading all 64 at once would cause an out-of-memory error.

How Gradient Accumulation Works

Instead of computing gradients on all 64 examples simultaneously, gradient accumulation breaks them into smaller mini-batches (say, 4 examples each). Here's the key insight:

  1. Forward pass on mini-batch 1 → compute loss
  2. Backward pass → compute gradients, but don't update weights yet
  3. Accumulate (add) these gradients to a running total
  4. Repeat steps 1-3 for mini-batches 2, 3, ..., 16
  5. After all mini-batches, average the accumulated gradients
  6. Update model weights once with the averaged gradient

The Analogy

Think of it like taking multiple measurements to reduce noise. Rather than measuring once with a big ruler you can't fit, you measure 16 times with a small ruler and average the results—mathematically equivalent, but memory-friendly.

Why It's Critical

  • Memory efficiency: Train with effective batch size of 64 while only using memory for 4
  • Gradient stability: Larger effective batches mean less noisy gradient estimates
  • Distributed training: Essential when combined with techniques like DDP or FSDP to maximize throughput without exceeding memory

The tradeoff? More computation time per weight update, since you're doing multiple forward/backward passes.

Key Takeaway: Gradient accumulation simulates large-batch training by summing gradients across multiple small mini-batches before updating weights, enabling memory-constrained training of large models.