Course contentsShow
← Machine Learning and Deep Learning
Lesson 2739 of 3,53859. Distributed Training: Data ParallelismFree lesson

Activation Checkpointing with FSDP

Combining gradient checkpointing and FSDP to trade recomputation for memory when training very large models.

Activation Checkpointing with FSDP

What You'll Learn

You'll learn how to combine activation checkpointing (gradient checkpointing) with FSDP to dramatically reduce peak memory usage when training massive models, at the cost of extra computation.

The Memory-Compute Trade-off

When training enormous models with FSDP, even though parameters and gradients are sharded across devices, activations can still consume huge amounts of memory. Remember, activations are the intermediate outputs stored during the forward pass—needed later for backpropagation.

Activation checkpointing solves this by not storing most intermediate activations. Instead, it only saves activations at certain "checkpoint" boundaries. During the backward pass, it recomputes the missing activations on-the-fly by re-running forward passes for those segments.

Think of it like taking notes during a lecture: without checkpointing, you record everything (high memory). With checkpointing, you only write down key section headers, then reconstruct details by rewatching those sections later (less memory, more time).

Why Combine with FSDP?

FSDP already shards model parameters, but activations are typically not sharded—each GPU holds full activation tensors for its local batch. For very large models or batch sizes, activations can become the dominant memory consumer.

By applying activation checkpointing on top of FSDP, you:

  • Save checkpoints only at transformer block boundaries (or similar module boundaries)
  • Recompute activations during backward pass
  • Trade ~33% extra compute time for potentially 50%+ memory savings

This enables training models that simply wouldn't fit otherwise, even with FSDP's parameter sharding.

Practical Implementation Pattern

In PyTorch FSDP, you typically apply checkpointing to wrapped submodules (like individual transformer layers), ensuring checkpointed regions align with FSDP wrapping boundaries for optimal memory savings.

Key Takeaway: Activation checkpointing with FSDP trades recomputation cost for memory savings, enabling training of models too large to fit even with parameter sharding alone—essential for frontier-scale model development.