Combining Gradient Accumulation and Checkpointing
What you'll learn: How to use gradient accumulation and activation checkpointing together to train models that would otherwise exceed your memory budget.
The Power of Two Techniques Together
You've learned two complementary memory optimization strategies: gradient accumulation reduces memory by processing smaller physical batches while maintaining a large effective batch size, and activation checkpointing trades computation for memory by recomputing activations during backward pass instead of storing them all.
When combined, these techniques multiply your capabilities. Gradient accumulation lets you use tiny micro-batches that fit in memory, while checkpointing reduces the memory footprint of each micro-batch itself. This synergy enables training models that would be impossible with either technique alone.
Configuration Best Practices
The key is finding the right balance:
- Start with checkpointing to reduce per-batch memory usage
- Add gradient accumulation to reach your desired effective batch size
- Monitor memory closely — aim for 80-90% GPU utilization without OOM errors
# Combined implementation
accumulation_steps = 8
checkpoint_segments = 4
for batch_idx, (inputs, targets) in enumerate(dataloader):
# Checkpointing reduces memory per forward pass
outputs = checkpoint_sequential(model, checkpoint_segments, inputs)
loss = criterion(outputs, targets) / accumulation_steps
loss.backward() # Gradients accumulate
if (batch_idx + 1) % accumulation_steps == 0:
optimizer.step()
optimizer.zero_grad()
Trade-off considerations: More accumulation steps increase training time linearly, while more checkpoint segments increase backward pass time (typically 20-30% overhead). Tune both to maximize throughput while staying within memory limits.
Key Takeaway: Gradient accumulation and activation checkpointing are complementary memory optimizations that, when combined strategically, enable training large models on limited hardware by reducing both per-sample and per-batch memory requirements.