Course contentsShow
← Machine Learning and Deep Learning
Lesson 638 of 3,53815. Neural Network FoundationsFree lesson

Memory Requirements of Backpropagation

Analyzing memory costs of storing activations and how it scales with network depth and batch size.

Memory Requirements of Backpropagation

What You'll Learn

You'll understand why training neural networks requires significantly more memory than just running inference, and how network depth and batch size directly impact memory consumption.

The Memory Challenge

When you perform a forward pass, the network computes activations layer by layer. But here's the catch: to compute gradients during backpropagation, you need to store all the intermediate activations from the forward pass. Why? Because the chain rule requires these values to calculate how each weight contributed to the final loss.

Think of it like retracing your steps through a maze. You can't figure out which turns were wrong unless you remember every turn you made.

Memory Scaling Factors

Network Depth

The deeper your network, the more layers of activations you must store. A 10-layer network requires storing activations from all 10 layers simultaneously during training. This creates a linear relationship between depth and memory usage.

Batch Size

Processing multiple examples together (batching) multiplies memory requirements. If one example's activations take 100MB and you use a batch size of 32, you need roughly 3.2GB just for activations. Memory scales linearly with batch size.

Hidden Layer Width

Wider layers (more neurons per layer) mean larger activation matrices. A layer with 1,000 neurons versus 100 neurons stores 10× more values.

Why This Matters

This memory bottleneck is why:

  • You can't always use arbitrarily large batch sizes
  • Very deep networks require specialized hardware (GPUs with large memory)
  • Gradient checkpointing techniques trade computation for memory by recomputing some activations

Key Takeaway: Backpropagation requires storing all forward pass activations, causing memory to scale linearly with network depth, batch size, and layer width—making memory management a critical consideration in deep learning.