Implementing Activation Checkpointing in PyTorch
You'll learn how to use torch.utils.checkpoint to selectively recompute activations during backpropagation, trading compute for memory in specific model layers.
The Core Idea
Activation checkpointing (also called gradient checkpointing) doesn't store intermediate activations during the forward pass for selected layers. Instead, it recomputes them on-demand during the backward pass. This is like taking notes during a lecture: you can either write down everything (use lots of paper/memory) or write down only key points and reconstruct details later (use more time/compute).
Using torch.utils.checkpoint
PyTorch provides torch.utils.checkpoint.checkpoint() to wrap model segments. When you checkpoint a function, PyTorch:
- Runs the forward pass normally but doesn't save intermediate activations
- During backpropagation, re-runs the forward pass to recompute those activations
- Computes gradients with the recomputed values
import torch
from torch.utils.checkpoint import checkpoint
class CheckpointedTransformerBlock(torch.nn.Module):
def __init__(self, block):
super().__init__()
self.block = block
def forward(self, x):
# Use checkpointing for this block
return checkpoint(self.block, x, use_reentrant=False)
# Apply to specific layers
model = TransformerModel()
model.layers[5] = CheckpointedTransformerBlock(model.layers[5])
model.layers[10] = CheckpointedTransformerBlock(model.layers[10])
The use_reentrant=False flag is recommended for modern PyTorch versions as it provides better compatibility with distributed training patterns.
You typically checkpoint expensive layers like transformer blocks or residual blocks where activation memory is large. Don't checkpoint every layer—that wastes compute without proportional memory benefit.
Key Takeaway: torch.utils.checkpoint lets you strategically recompute activations for memory-heavy layers, reducing peak memory at the cost of ~33% extra compute per checkpointed segment.