Course contentsShow
← Machine Learning and Deep Learning
Lesson 1688 of 3,53836. LLM Inference OptimizationFree lesson

Activation Checkpointing for Attention

Trading compute for memory by recomputing attention activations during backward pass instead of storing them.

Activation Checkpointing for Attention

What you'll learn: How to dramatically reduce memory usage during training by recomputing attention activations instead of storing them all.

The Memory Problem During Training

During the backward pass of training, neural networks need the forward pass activations to compute gradients. For attention mechanisms, this means storing every intermediate result from the attention computation—query-key products, softmax outputs, attention weights, and value combinations. With long sequences and many layers, this memory requirement explodes.

Trading Compute for Memory

Activation checkpointing (also called gradient checkpointing) solves this by discarding most intermediate activations during the forward pass, keeping only strategic "checkpoints." During the backward pass, when those activations are needed to compute gradients, you simply recompute them on-the-fly from the nearest checkpoint.

Think of it like studying for an exam: instead of writing down every detail from every lecture (storing everything), you keep summary notes at key points (checkpoints) and can re-derive the details when needed (recompute).

How It Works for Attention

For attention layers specifically:

  1. Forward pass: Compute attention normally, but only save the layer's input and output—discard intermediate attention matrices
  2. Backward pass: When gradients are needed, re-run the forward computation for that attention block to regenerate the intermediate values
  3. Use those freshly computed activations to calculate gradients
  4. Discard them again after use

The Trade-off

You're essentially performing the forward pass twice—once during the actual forward pass, and again during backpropagation. This adds ~33% computational overhead but can reduce memory usage by 50% or more, enabling you to train with larger batch sizes or longer context windows within the same GPU memory budget.

Key Takeaway: Activation checkpointing lets you train larger models or use longer sequences by recomputing attention activations during backprop instead of storing them, trading extra computation for significant memory savings.