Gradient Checkpointing and Memory Trade-offs
What you'll learn: How to dramatically reduce memory usage during neural network training by selectively recomputing activations instead of storing them all.
The Memory Problem in Deep Networks
During backpropagation, you need access to intermediate activations from the forward pass to compute gradients. As we learned in "Memory Requirements of Backpropagation," storing all these activations for deep networks can consume enormous amounts of memory—often the primary bottleneck in training large models.
What is Gradient Checkpointing?
Gradient checkpointing is a clever memory-saving technique: instead of storing all intermediate activations from the forward pass, you only store a few checkpoints at selected layers. During backpropagation, when you need an activation that wasn't saved, you recompute it on-the-fly by running a mini forward pass from the nearest checkpoint.
Think of it like hiking a mountain: you could take photos at every step (high memory), or just mark a few key spots and retrace your steps when needed (less memory, more walking).
The Trade-off
Memory savings: You might store only 10-20% of activations, enabling training of much larger models or bigger batch sizes.
Computation cost: You effectively run the forward pass roughly 1.5-2× instead of just once, because you're recomputing discarded activations during backpropagation.
This is particularly valuable when memory is your limiting factor (GPUs with limited VRAM) and you have computation cycles to spare.
When to Use It
Gradient checkpointing shines when:
- Training very deep networks that won't fit in memory otherwise
- You want to increase batch size for better gradient estimates
- Memory is more constrained than computation time
Key Takeaway: Gradient checkpointing trades computation for memory by selectively recomputing intermediate activations during the backward pass, enabling training of larger models at the cost of ~50% more forward computation time.