Course contentsShow
← Machine Learning and Deep Learning
Lesson 628 of 3,53815. Neural Network FoundationsFree lesson

Loss Function Gradient: Starting Backpropagation

Computing the initial gradient of the loss with respect to network outputs as the first step of backpropagation.

Loss Function Gradient: Starting Backpropagation

What you'll learn: How to compute the very first gradient—the derivative of the loss with respect to the network's output—to kickstart the backpropagation process.

The Starting Point of Learning

You've seen how forward propagation pushes inputs through the network to produce outputs, and how the loss function measures how wrong those outputs are. Now we need to improve the network. Backpropagation works backward from the loss, but it needs somewhere to begin.

The first step is computing how the loss changes with respect to the network's final output. This initial gradient tells us: "If I nudge the network's prediction slightly in this direction, how much does my error increase or decrease?"

Computing the Initial Gradient

For a network output ŷ and true label y, if you're using:

  • Mean Squared Error: The gradient is ∂L/∂ŷ = 2(ŷ - y) / n
  • Binary Cross-Entropy: The gradient is ∂L/∂ŷ = (ŷ - y) / (ŷ(1 - ŷ))
  • Categorical Cross-Entropy (with softmax): The gradient simplifies to ∂L/∂ŷ = ŷ - y

Notice these formulas depend on both the prediction and the true label. The gradient captures the direction and magnitude of error.

Why This Matters

Think of this gradient as the "error signal" that will flow backward through your network. Just like dropping a pebble creates ripples that spread outward on water, this initial gradient will propagate backward through every layer using the chain rule you learned earlier. Each layer will use this signal to figure out how to adjust its own weights.

Without this starting gradient, backpropagation has nowhere to begin—it's the spark that ignites the entire learning process.

Key Takeaway: The loss function gradient with respect to network outputs is the initial error signal that launches backpropagation, computed directly from the prediction error and the derivative of your chosen loss function.