Activation Checkpointing Fundamentals
What you'll learn: How to dramatically reduce memory usage during training by strategically recomputing activations instead of storing them all.
The Memory Problem
During neural network training's forward pass, you compute and store activations at every layer—you need these to calculate gradients during the backward pass. For deep models like transformers with billions of parameters, storing all these intermediate activations can consume enormous amounts of GPU memory, often exceeding what's available.
The Core Trade-off
Activation checkpointing (also called gradient checkpointing) solves this by making a deliberate trade-off: store less, compute more.
Instead of keeping all activations in memory, you:
- Forward pass: Only save activations at selected "checkpoint" layers (e.g., every Nth layer)
- Backward pass: When you need activations for layers between checkpoints, recompute them on-the-fly by doing mini forward passes from the nearest checkpoint
Think of it like hiking with limited backpack space: instead of carrying water for the entire trail (storing all activations), you carry a filter and refill from streams along the way (recompute from checkpoints).
Why It Works
Modern GPUs are often memory-bound rather than compute-bound for large models. The extra computation cost (typically 20-33% more FLOPs) is acceptable because:
- You can fit larger models or batch sizes that wouldn't fit otherwise
- The recomputation happens in parallel with other operations
- Memory freed enables training that was previously impossible
This technique is especially powerful when combined with other memory optimization strategies like FSDP or ZeRO.
Key Takeaway: Activation checkpointing trades additional forward-pass computation for substantial memory savings, enabling training of larger models by selectively discarding and recomputing intermediate activations during backpropagation.