Course contentsShow
← Machine Learning and Deep Learning
Lesson 649 of 3,53815. Neural Network FoundationsFree lesson

Gradient Checkpointing and Memory Trade-offs

Recompute intermediate activations during backward pass to reduce memory usage at the cost of computation time.

Gradient Checkpointing and Memory Trade-offs

What you'll learn: How to dramatically reduce memory usage during neural network training by selectively recomputing activations instead of storing them all.

The Memory Problem in Deep Networks

During backpropagation, you need access to intermediate activations from the forward pass to compute gradients. As we learned in "Memory Requirements of Backpropagation," storing all these activations for deep networks can consume enormous amounts of memory—often the primary bottleneck in training large models.

What is Gradient Checkpointing?

Gradient checkpointing is a clever memory-saving technique: instead of storing all intermediate activations from the forward pass, you only store a few checkpoints at selected layers. During backpropagation, when you need an activation that wasn't saved, you recompute it on-the-fly by running a mini forward pass from the nearest checkpoint.

Think of it like hiking a mountain: you could take photos at every step (high memory), or just mark a few key spots and retrace your steps when needed (less memory, more walking).

The Trade-off

Memory savings: You might store only 10-20% of activations, enabling training of much larger models or bigger batch sizes.

Computation cost: You effectively run the forward pass roughly 1.5-2× instead of just once, because you're recomputing discarded activations during backpropagation.

This is particularly valuable when memory is your limiting factor (GPUs with limited VRAM) and you have computation cycles to spare.

When to Use It

Gradient checkpointing shines when:

  • Training very deep networks that won't fit in memory otherwise
  • You want to increase batch size for better gradient estimates
  • Memory is more constrained than computation time

Key Takeaway: Gradient checkpointing trades computation for memory by selectively recomputing intermediate activations during the backward pass, enabling training of larger models at the cost of ~50% more forward computation time.