Output Layer Gradient Derivation
You'll learn how to derive the gradients for weights and biases in the final layer by applying the chain rule to connect loss, activation, and layer inputs.
The Core Idea
When backpropagation begins, we start at the output layer where the loss function is computed. To update the weights and biases that produced this output, we need to calculate how much each parameter contributed to the loss. This is where the chain rule becomes essential.
Think of it like tracing blame backwards: if your prediction was wrong, you need to figure out how much each weight and bias in that final layer was responsible for the error.
Breaking Down the Gradient
For the output layer, we compute two key gradients:
- Gradient with respect to weights: How does changing each weight affect the loss?
- Gradient with respect to biases: How does changing each bias affect the loss?
Using the chain rule, the weight gradient is:
∂Loss/∂W_output = ∂Loss/∂activation × ∂activation/∂z × ∂z/∂W_output
Where z is the pre-activation (weighted sum before applying the activation function).
For the bias:
∂Loss/∂b_output = ∂Loss/∂activation × ∂activation/∂z
Each piece is straightforward: the loss gradient tells us how wrong we are, the activation derivative shows how sensitive the activation is, and the input derivative links it to the actual parameters.
Key Takeaway: Output layer gradients are computed by chaining together three derivatives: loss with respect to activation, activation with respect to pre-activation, and pre-activation with respect to weights/biases. This gives us the exact direction to adjust parameters.