Course contentsShow
← Machine Learning and Deep Learning
Lesson 2786 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

Activation Checkpointing Fundamentals

Trading computation for memory by recomputing activations during backward pass instead of storing them from forward pass.

Activation Checkpointing Fundamentals

What you'll learn: How to dramatically reduce memory usage during training by strategically recomputing activations instead of storing them all.

The Memory Problem

During neural network training's forward pass, you compute and store activations at every layer—you need these to calculate gradients during the backward pass. For deep models like transformers with billions of parameters, storing all these intermediate activations can consume enormous amounts of GPU memory, often exceeding what's available.

The Core Trade-off

Activation checkpointing (also called gradient checkpointing) solves this by making a deliberate trade-off: store less, compute more.

Instead of keeping all activations in memory, you:

  1. Forward pass: Only save activations at selected "checkpoint" layers (e.g., every Nth layer)
  2. Backward pass: When you need activations for layers between checkpoints, recompute them on-the-fly by doing mini forward passes from the nearest checkpoint

Think of it like hiking with limited backpack space: instead of carrying water for the entire trail (storing all activations), you carry a filter and refill from streams along the way (recompute from checkpoints).

Why It Works

Modern GPUs are often memory-bound rather than compute-bound for large models. The extra computation cost (typically 20-33% more FLOPs) is acceptable because:

  • You can fit larger models or batch sizes that wouldn't fit otherwise
  • The recomputation happens in parallel with other operations
  • Memory freed enables training that was previously impossible

This technique is especially powerful when combined with other memory optimization strategies like FSDP or ZeRO.

Key Takeaway: Activation checkpointing trades additional forward-pass computation for substantial memory savings, enabling training of larger models by selectively discarding and recomputing intermediate activations during backpropagation.