Course contentsShow
← Machine Learning and Deep Learning
Lesson 2767 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

Memory Footprint Analysis

Calculate memory requirements for activations, gradients, and parameters under different parallelism strategies.

Memory Footprint Analysis

What you'll learn: How to calculate and predict the memory requirements for activations, gradients, and parameters under different distributed training strategies.

Why Memory Footprint Matters

When training large models across multiple GPUs, understanding exactly where your memory goes is critical. Run out of memory mid-training, and your job crashes. Underestimate requirements, and you can't scale to the model size you need. Memory footprint analysis helps you plan capacity and choose the right parallelism strategy.

The Three Memory Components

Think of GPU memory like a warehouse with three storage areas:

1. Parameters (Model Weights)

The actual weights of your model. For a model with P parameters stored in float32, you need 4P bytes. With float16/bfloat16, this drops to 2P bytes.

2. Gradients

During backpropagation, you store one gradient per parameter. Same size as parameters: 4P bytes (fp32) or 2P bytes (fp16).

3. Activations (Intermediate Outputs)

The forward pass outputs you must keep for backpropagation. Size depends on batch size B, sequence length L, hidden dimension H, and number of layers N. Roughly 2 × B × L × H × N bytes for Transformers.

Impact of Parallelism Strategies

  • Data Parallelism: Full model replica per GPU → each device holds all 3 components
  • FSDP/ZeRO Stage 3: Parameters and gradients sharded across K GPUs → divide by K
  • Pipeline Parallelism: Parameters divided by pipeline stages → activations reduced but microbatch management adds overhead
  • Tensor Parallelism: Parameters sharded within layers → all-reduce increases activation memory slightly
  • Activation Checkpointing: Trade ~40% recomputation time to save ~50-80% activation memory

Key Takeaway: Memory footprint = parameters + gradients + activations (+ optimizer states). Each parallelism strategy redistributes these components differently—calculate before you scale to avoid surprises.