Course contentsShow
← Machine Learning and Deep Learning
Lesson 608 of 3,53815. Neural Network FoundationsFree lesson

Intermediate Activations and Their Role

Understanding why we store intermediate layer outputs (activations) during forward propagation for later backpropagation.

Intermediate Activations and Their Role

What you'll learn: Why neural networks must save the output of each hidden layer during forward propagation, and how these stored values become essential for training.

What Are Intermediate Activations?

When you perform forward propagation through a neural network, each hidden layer produces an output—these outputs are called intermediate activations (or simply "activations"). They're called "intermediate" because they're not the final network output; they're the values computed between the input and the final prediction.

For example, in a 3-layer network:

  • Layer 1 produces activations a1
  • Layer 2 produces activations a2
  • Layer 3 produces the final output a3

Both a1 and a2 are intermediate activations.

Why Store Them?

Here's the crucial insight: you must save these intermediate activations during forward propagation because you'll need them later during backpropagation.

Think of it like climbing a mountain while dropping breadcrumbs. On the way up (forward pass), you mark your path. On the way down (backward pass), you follow those breadcrumbs to retrace your steps and figure out how steep each section was.

During backpropagation, you calculate gradients (derivatives) that tell you how to adjust weights. These gradient calculations require knowing what values flowed through each layer during the forward pass. Without storing activations, you'd have to recompute the entire forward pass again—wasteful and inefficient.

What Gets Stored

For each layer, you typically store:

  • The pre-activation values (before applying the activation function)
  • The post-activation values (after applying the activation function)

Both are needed because backpropagation chains derivatives through these values layer by layer.

Key Takeaway: Intermediate activations are the outputs of hidden layers during forward propagation that must be cached in memory because backpropagation needs them to compute gradients efficiently. They're the "memory" that makes learning possible.