Course contentsShow
← Machine Learning and Deep Learning
Lesson 2782 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

Implementing Gradient Accumulation in PyTorch

Step-by-step implementation of gradient accumulation loops, including when to zero gradients and scale loss properly.

Implementing Gradient Accumulation in PyTorch

You'll learn the exact mechanics of coding a gradient accumulation loop, including when to zero gradients and how to scale your loss.

The Core Implementation Pattern

Gradient accumulation splits one large batch into several smaller micro-batches, accumulating gradients across them before updating weights. The key is understanding three critical moments: when to zero gradients, when to scale loss, and when to step the optimizer.

Step-by-Step Loop Structure

Here's the canonical pattern:

optimizer.zero_grad()  # Zero ONCE before accumulation starts

for i, (inputs, targets) in enumerate(dataloader):
    outputs = model(inputs)
    loss = criterion(outputs, targets)
    
    # Scale loss by number of accumulation steps
    loss = loss / accumulation_steps
    
    loss.backward()  # Accumulate gradients
    
    # Update weights only every N steps
    if (i + 1) % accumulation_steps == 0:
        optimizer.step()
        optimizer.zero_grad()

Critical Details

When to zero gradients: Only after optimizer steps, not after every backward pass. Otherwise you'd erase the accumulated gradients!

Why scale the loss: Without dividing by accumulation_steps, your effective learning rate would be multiplied by that factor. The scaling ensures gradients have the correct magnitude, as if you'd processed one large batch.

Think of it like filling a bucket: Each micro-batch adds water (gradients). You only pour it out (optimizer step) when the bucket is full, then empty it (zero_grad) for the next accumulation cycle.

Key Takeaway: Gradient accumulation requires zeroing gradients only after optimizer steps, scaling loss by the number of accumulation steps to maintain correct gradient magnitudes, and stepping the optimizer only when all micro-batches have contributed their gradients.