Course contentsShow
← Machine Learning and Deep Learning
Lesson 2787 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

Implementing Activation Checkpointing in PyTorch

Using torch.utils.checkpoint to apply gradient checkpointing to specific layers or transformer blocks in your models.

Implementing Activation Checkpointing in PyTorch

You'll learn how to use torch.utils.checkpoint to selectively recompute activations during backpropagation, trading compute for memory in specific model layers.

The Core Idea

Activation checkpointing (also called gradient checkpointing) doesn't store intermediate activations during the forward pass for selected layers. Instead, it recomputes them on-demand during the backward pass. This is like taking notes during a lecture: you can either write down everything (use lots of paper/memory) or write down only key points and reconstruct details later (use more time/compute).

Using torch.utils.checkpoint

PyTorch provides torch.utils.checkpoint.checkpoint() to wrap model segments. When you checkpoint a function, PyTorch:

  1. Runs the forward pass normally but doesn't save intermediate activations
  2. During backpropagation, re-runs the forward pass to recompute those activations
  3. Computes gradients with the recomputed values
import torch
from torch.utils.checkpoint import checkpoint

class CheckpointedTransformerBlock(torch.nn.Module):
    def __init__(self, block):
        super().__init__()
        self.block = block
    
    def forward(self, x):
        # Use checkpointing for this block
        return checkpoint(self.block, x, use_reentrant=False)

# Apply to specific layers
model = TransformerModel()
model.layers[5] = CheckpointedTransformerBlock(model.layers[5])
model.layers[10] = CheckpointedTransformerBlock(model.layers[10])

The use_reentrant=False flag is recommended for modern PyTorch versions as it provides better compatibility with distributed training patterns.

You typically checkpoint expensive layers like transformer blocks or residual blocks where activation memory is large. Don't checkpoint every layer—that wastes compute without proportional memory benefit.

Key Takeaway: torch.utils.checkpoint lets you strategically recompute activations for memory-heavy layers, reducing peak memory at the cost of ~33% extra compute per checkpointed segment.