Selective Checkpointing Strategies
What you'll learn: How to strategically choose which layers to checkpoint to optimize the memory-computation tradeoff in large model training.
The Strategic Choice Problem
You've learned that activation checkpointing saves memory by recomputing activations during the backward pass instead of storing them all. But checkpointing everything wastes computation, while checkpointing nothing wastes memory. The solution? Selective checkpointing — intelligently choosing which layers to checkpoint based on their memory footprint and recomputation cost.
The Memory-Computation Tradeoff
Think of your model as a city with different neighborhoods. Some areas (transformer blocks, large convolutional layers) have massive "storage facilities" (activations) but are relatively quick to "rebuild." Others (simple operations like ReLU) take tiny storage but you'd need to checkpoint many of them to see real savings.
The winning strategy: Checkpoint the "heavy hitters" — typically:
- Transformer blocks (attention + feedforward sublayers together)
- Large linear/convolutional layers with high activation memory
- Entire residual blocks in CNNs
Skip checkpointing:
- Lightweight operations (normalization, activation functions alone)
- Early layers with small spatial dimensions
- Final layers close to the loss (minimal memory savings)
Practical Guidelines
Most practitioners checkpoint at coarse module boundaries rather than individual operations. For transformers, checkpointing every N-th transformer block (e.g., every 2nd or 3rd) provides an excellent balance. This approach is simple to implement and matches natural computational boundaries.
Key Takeaway: Selective checkpointing targets high-memory modules (like transformer blocks) while avoiding fine-grained checkpointing of cheap operations, achieving 2-3x memory reduction with minimal (<15%) computational overhead.