Course contentsShow
← Machine Learning and Deep Learning
Lesson 2790 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

Combining Gradient Accumulation and Checkpointing

Using both techniques together to train large models with limited memory, including best practices for configuration.

Combining Gradient Accumulation and Checkpointing

What you'll learn: How to use gradient accumulation and activation checkpointing together to train models that would otherwise exceed your memory budget.

The Power of Two Techniques Together

You've learned two complementary memory optimization strategies: gradient accumulation reduces memory by processing smaller physical batches while maintaining a large effective batch size, and activation checkpointing trades computation for memory by recomputing activations during backward pass instead of storing them all.

When combined, these techniques multiply your capabilities. Gradient accumulation lets you use tiny micro-batches that fit in memory, while checkpointing reduces the memory footprint of each micro-batch itself. This synergy enables training models that would be impossible with either technique alone.

Configuration Best Practices

The key is finding the right balance:

  1. Start with checkpointing to reduce per-batch memory usage
  2. Add gradient accumulation to reach your desired effective batch size
  3. Monitor memory closely — aim for 80-90% GPU utilization without OOM errors
# Combined implementation
accumulation_steps = 8
checkpoint_segments = 4

for batch_idx, (inputs, targets) in enumerate(dataloader):
    # Checkpointing reduces memory per forward pass
    outputs = checkpoint_sequential(model, checkpoint_segments, inputs)
    loss = criterion(outputs, targets) / accumulation_steps
    
    loss.backward()  # Gradients accumulate
    
    if (batch_idx + 1) % accumulation_steps == 0:
        optimizer.step()
        optimizer.zero_grad()

Trade-off considerations: More accumulation steps increase training time linearly, while more checkpoint segments increase backward pass time (typically 20-30% overhead). Tune both to maximize throughput while staying within memory limits.

Key Takeaway: Gradient accumulation and activation checkpointing are complementary memory optimizations that, when combined strategically, enable training large models on limited hardware by reducing both per-sample and per-batch memory requirements.