Course contentsShow
← Machine Learning and Deep Learning
Lesson 629 of 3,53815. Neural Network FoundationsFree lesson

Output Layer Gradient Derivation

Deriving gradients for weights and biases in the final layer using chain rule on loss and activation functions.

Output Layer Gradient Derivation

You'll learn how to derive the gradients for weights and biases in the final layer by applying the chain rule to connect loss, activation, and layer inputs.

The Core Idea

When backpropagation begins, we start at the output layer where the loss function is computed. To update the weights and biases that produced this output, we need to calculate how much each parameter contributed to the loss. This is where the chain rule becomes essential.

Think of it like tracing blame backwards: if your prediction was wrong, you need to figure out how much each weight and bias in that final layer was responsible for the error.

Breaking Down the Gradient

For the output layer, we compute two key gradients:

  1. Gradient with respect to weights: How does changing each weight affect the loss?
  2. Gradient with respect to biases: How does changing each bias affect the loss?

Using the chain rule, the weight gradient is:

∂Loss/∂W_output = ∂Loss/∂activation × ∂activation/∂z × ∂z/∂W_output

Where z is the pre-activation (weighted sum before applying the activation function).

For the bias:

∂Loss/∂b_output = ∂Loss/∂activation × ∂activation/∂z

Each piece is straightforward: the loss gradient tells us how wrong we are, the activation derivative shows how sensitive the activation is, and the input derivative links it to the actual parameters.

Key Takeaway: Output layer gradients are computed by chaining together three derivatives: loss with respect to activation, activation with respect to pre-activation, and pre-activation with respect to weights/biases. This gives us the exact direction to adjust parameters.