Memory Requirements of Backpropagation
What You'll Learn
You'll understand why training neural networks requires significantly more memory than just running inference, and how network depth and batch size directly impact memory consumption.
The Memory Challenge
When you perform a forward pass, the network computes activations layer by layer. But here's the catch: to compute gradients during backpropagation, you need to store all the intermediate activations from the forward pass. Why? Because the chain rule requires these values to calculate how each weight contributed to the final loss.
Think of it like retracing your steps through a maze. You can't figure out which turns were wrong unless you remember every turn you made.
Memory Scaling Factors
Network Depth
The deeper your network, the more layers of activations you must store. A 10-layer network requires storing activations from all 10 layers simultaneously during training. This creates a linear relationship between depth and memory usage.
Batch Size
Processing multiple examples together (batching) multiplies memory requirements. If one example's activations take 100MB and you use a batch size of 32, you need roughly 3.2GB just for activations. Memory scales linearly with batch size.
Hidden Layer Width
Wider layers (more neurons per layer) mean larger activation matrices. A layer with 1,000 neurons versus 100 neurons stores 10× more values.
Why This Matters
This memory bottleneck is why:
- You can't always use arbitrarily large batch sizes
- Very deep networks require specialized hardware (GPUs with large memory)
- Gradient checkpointing techniques trade computation for memory by recomputing some activations
Key Takeaway: Backpropagation requires storing all forward pass activations, causing memory to scale linearly with network depth, batch size, and layer width—making memory management a critical consideration in deep learning.