Course contentsShow
← Machine Learning and Deep Learning
Lesson 794 of 3,53819. PyTorch FundamentalsFree lesson

Gradient Accumulation and .zero_grad()

Why gradients accumulate by default and how to clear them between training iterations using zero_grad().

Gradient Accumulation and .zero_grad()

You'll learn why PyTorch accumulates gradients by default and how to prevent errors by clearing them between training steps.

The Accumulation Problem

When you call .backward() on a loss, PyTorch computes gradients and stores them in each parameter's .grad attribute. Here's the crucial detail: PyTorch adds new gradients to whatever is already in .grad rather than replacing them.

This behavior is intentional—it allows advanced techniques like gradient accumulation across multiple mini-batches. But for typical training, it causes a problem: if you don't clear old gradients, they keep piling up like unpaid bills, making your optimizer update parameters using the sum of gradients from multiple iterations instead of just the current one.

The Solution: zero_grad()

Before computing gradients for a new batch, you must explicitly zero out the accumulated gradients:

optimizer.zero_grad()  # Clear old gradients
loss.backward()         # Compute new gradients
optimizer.step()        # Update parameters

Think of .zero_grad() as wiping a whiteboard clean before solving a new problem. Without it, you're writing over old work, creating a confusing mess.

Why Accumulation Exists

Accumulation is useful when your batch size is limited by memory. You can:

  1. Process several small batches
  2. Let gradients accumulate (don't call .zero_grad())
  3. Update once with the accumulated gradient (simulating a larger batch)

But for standard single-batch training, always call .zero_grad() at the start of each iteration.

Key Takeaway: PyTorch accumulates gradients by default—always call optimizer.zero_grad() before computing new gradients to prevent incorrect parameter updates from summed historical gradients.