Hidden Layer Gradient: Recursive Application
What you'll learn: How to compute gradients for hidden layers by recursively propagating errors backward through the network.
The Core Challenge
You've learned how to compute gradients for the output layer—that's where you have direct access to the loss. But what about hidden layers? They don't "see" the loss directly. The solution: propagate the error backward through each layer, one at a time, using the chain rule recursively.
How It Works
Think of it like a relay race where blame is passed backward. Each hidden layer asks: "How much did I contribute to the final error?" The answer comes from the layer after it.
For a hidden layer with activation h, the gradient flows through two components:
- The layer ahead sends back its gradient (how sensitive the loss is to this layer's output)
- The activation function scales that gradient based on how much it "gates" information
Mathematically, for hidden layer l:
∂Loss/∂h_l = (∂Loss/∂h_(l+1)) × (∂h_(l+1)/∂h_l)
This is recursive: you need the gradient from layer l+1 to compute the gradient for layer l. You start at the output (where you computed the loss gradient) and work backward, layer by layer.
Each step involves:
- Taking the gradient from the next layer
- Multiplying by the derivative of the current activation function
- Multiplying by the weight matrix (since weights connect layers)
Why "Recursive"?
You're applying the same pattern at each layer, just with different intermediate values. Layer 3 needs gradients from layer 4. Layer 2 needs gradients from layer 3. And so on, all the way back to layer 1.
Key Takeaway: Hidden layer gradients are computed recursively by backpropagating errors from deeper layers, multiplying through activation derivatives and weights at each step—the chain rule applied layer by layer.