Memory Savings vs Computational Overhead
You'll learn: How to quantify the memory-computation tradeoff when using activation checkpointing in distributed training.
The Core Tradeoff
Activation checkpointing saves memory by discarding intermediate activations during the forward pass, then recomputing them during backpropagation. This is a deliberate time-for-space tradeoff: you trade additional compute for reduced memory footprint.
The typical cost structure:
- Memory reduction: 30-80% depending on which layers you checkpoint
- Computational overhead: ~30% additional training time from recomputation
Think of it like a warehouse that ships products without keeping inventory records. To know what's in stock during audit time (backward pass), workers must physically recount everything—trading labor time for storage space.
Quantifying the Tradeoff
When deciding whether to use checkpointing:
Memory savings = (original activation memory) - (checkpoint memory) - (recomputed activation memory during backward)
For a typical transformer layer storing all activations might require 4GB, while checkpointing only boundary activations might use 0.5GB—an 87% reduction.
Computational cost comes from re-running forward operations during the backward pass. If your forward pass takes 100ms, expect backward to increase from ~200ms to ~230ms (the extra 30ms is recomputation).
When It's Worth It
Checkpointing makes sense when:
- You're memory-bound (can't fit the model otherwise)
- The 30% slowdown is acceptable compared to smaller batch sizes or model sizes
- You're training very deep networks where activation memory dominates
It's less valuable when you're already compute-bound or have abundant GPU memory.
Key Takeaway: Activation checkpointing typically saves 30-80% of activation memory at the cost of ~30% additional computation—a worthwhile tradeoff when memory is the bottleneck preventing larger models or batch sizes.