Zero Gradients and Gradient Accumulation
What you'll learn: Why calling optimizer.zero_grad() before each backward pass prevents gradients from adding up incorrectly.
The Core Problem: Gradients Accumulate by Default
When you call .backward() on a loss tensor, PyTorch computes gradients and adds them to any existing gradients in the .grad attributes of your parameters. This is by design—PyTorch doesn't automatically clear old gradients.
Why? Because gradient accumulation can be useful (more on that next), but it's your responsibility to manage when gradients should be zeroed.
Why optimizer.zero_grad() Is Necessary
Think of gradients like notes on a whiteboard. Each .backward() call writes new notes, but PyTorch doesn't erase the old ones first. If you don't manually erase (zero) them, your next batch's gradients will add to the previous batch's gradients, giving you completely wrong gradient values.
Typical pattern:
for inputs, targets in dataloader:
optimizer.zero_grad() # Clear old gradients
outputs = model(inputs)
loss = criterion(outputs, targets)
loss.backward() # Compute new gradients (adds to .grad)
optimizer.step() # Update weights
When to Zero: Before or After?
Zero before .backward() (as shown above) is standard and clearest. Zeroing after optimizer.step() also works, but it's less common and can cause confusion if your code exits mid-loop with stale gradients.
Intentional Gradient Accumulation
Sometimes you want gradients to accumulate—for example, when simulating larger batch sizes on limited memory. You'd call .backward() multiple times before zeroing and stepping. But this is deliberate, not accidental.
Key Takeaway: Always call optimizer.zero_grad() before .backward() to clear old gradients unless you specifically intend to accumulate them across multiple batches.