Course contentsShow
← Machine Learning and Deep Learning
Lesson 2789 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

Memory Savings vs Computational Overhead

Quantifying the memory reduction from checkpointing against the ~30% computational cost of recomputing activations.

Memory Savings vs Computational Overhead

You'll learn: How to quantify the memory-computation tradeoff when using activation checkpointing in distributed training.

The Core Tradeoff

Activation checkpointing saves memory by discarding intermediate activations during the forward pass, then recomputing them during backpropagation. This is a deliberate time-for-space tradeoff: you trade additional compute for reduced memory footprint.

The typical cost structure:

  • Memory reduction: 30-80% depending on which layers you checkpoint
  • Computational overhead: ~30% additional training time from recomputation

Think of it like a warehouse that ships products without keeping inventory records. To know what's in stock during audit time (backward pass), workers must physically recount everything—trading labor time for storage space.

Quantifying the Tradeoff

When deciding whether to use checkpointing:

Memory savings = (original activation memory) - (checkpoint memory) - (recomputed activation memory during backward)

For a typical transformer layer storing all activations might require 4GB, while checkpointing only boundary activations might use 0.5GB—an 87% reduction.

Computational cost comes from re-running forward operations during the backward pass. If your forward pass takes 100ms, expect backward to increase from ~200ms to ~230ms (the extra 30ms is recomputation).

When It's Worth It

Checkpointing makes sense when:

  • You're memory-bound (can't fit the model otherwise)
  • The 30% slowdown is acceptable compared to smaller batch sizes or model sizes
  • You're training very deep networks where activation memory dominates

It's less valuable when you're already compute-bound or have abundant GPU memory.

Key Takeaway: Activation checkpointing typically saves 30-80% of activation memory at the cost of ~30% additional computation—a worthwhile tradeoff when memory is the bottleneck preventing larger models or batch sizes.