Memory Footprint Analysis
What you'll learn: How to calculate and predict the memory requirements for activations, gradients, and parameters under different distributed training strategies.
Why Memory Footprint Matters
When training large models across multiple GPUs, understanding exactly where your memory goes is critical. Run out of memory mid-training, and your job crashes. Underestimate requirements, and you can't scale to the model size you need. Memory footprint analysis helps you plan capacity and choose the right parallelism strategy.
The Three Memory Components
Think of GPU memory like a warehouse with three storage areas:
1. Parameters (Model Weights)
The actual weights of your model. For a model with P parameters stored in float32, you need 4P bytes. With float16/bfloat16, this drops to 2P bytes.
2. Gradients
During backpropagation, you store one gradient per parameter. Same size as parameters: 4P bytes (fp32) or 2P bytes (fp16).
3. Activations (Intermediate Outputs)
The forward pass outputs you must keep for backpropagation. Size depends on batch size B, sequence length L, hidden dimension H, and number of layers N. Roughly 2 × B × L × H × N bytes for Transformers.
Impact of Parallelism Strategies
- Data Parallelism: Full model replica per GPU → each device holds all 3 components
- FSDP/ZeRO Stage 3: Parameters and gradients sharded across K GPUs → divide by K
- Pipeline Parallelism: Parameters divided by pipeline stages → activations reduced but microbatch management adds overhead
- Tensor Parallelism: Parameters sharded within layers → all-reduce increases activation memory slightly
- Activation Checkpointing: Trade ~40% recomputation time to save ~50-80% activation memory
Key Takeaway: Memory footprint = parameters + gradients + activations (+ optimizer states). Each parallelism strategy redistributes these components differently—calculate before you scale to avoid surprises.