Intermediate Activations and Their Role
What you'll learn: Why neural networks must save the output of each hidden layer during forward propagation, and how these stored values become essential for training.
What Are Intermediate Activations?
When you perform forward propagation through a neural network, each hidden layer produces an output—these outputs are called intermediate activations (or simply "activations"). They're called "intermediate" because they're not the final network output; they're the values computed between the input and the final prediction.
For example, in a 3-layer network:
- Layer 1 produces activations
a1 - Layer 2 produces activations
a2 - Layer 3 produces the final output
a3
Both a1 and a2 are intermediate activations.
Why Store Them?
Here's the crucial insight: you must save these intermediate activations during forward propagation because you'll need them later during backpropagation.
Think of it like climbing a mountain while dropping breadcrumbs. On the way up (forward pass), you mark your path. On the way down (backward pass), you follow those breadcrumbs to retrace your steps and figure out how steep each section was.
During backpropagation, you calculate gradients (derivatives) that tell you how to adjust weights. These gradient calculations require knowing what values flowed through each layer during the forward pass. Without storing activations, you'd have to recompute the entire forward pass again—wasteful and inefficient.
What Gets Stored
For each layer, you typically store:
- The pre-activation values (before applying the activation function)
- The post-activation values (after applying the activation function)
Both are needed because backpropagation chains derivatives through these values layer by layer.
Key Takeaway: Intermediate activations are the outputs of hidden layers during forward propagation that must be cached in memory because backpropagation needs them to compute gradients efficiently. They're the "memory" that makes learning possible.