The Backward Pass Algorithm
What you'll learn: How backpropagation systematically computes gradients by traversing the network from output to input, layer by layer in reverse order.
The Complete Algorithm
You've seen the pieces—now let's assemble them into the complete backward pass algorithm. Backpropagation is essentially a structured reverse traversal of your neural network, computing gradients for each layer's parameters as you go.
The Step-by-Step Process
The algorithm follows this pattern:
-
Start at the loss: Compute the gradient of the loss function with respect to the final layer's output (∂Loss/∂output)
-
Work backward through layers: For each layer from last to first:
- Compute gradient with respect to the layer's weights and biases
- Compute gradient with respect to the layer's input (becomes the "gradient signal" for the previous layer)
-
Apply the chain rule repeatedly: Each layer's gradient depends on the gradient flowing back from layers ahead of it
-
Store all gradients: Collect weight and bias gradients for every layer—these will be used for parameter updates
The Reverse Order Intuition
Think of it like tracing blame for a mistake: if your final prediction is wrong, you first check the last decision maker (output layer), then ask "who influenced them?" (previous hidden layer), and continue backward until you've identified how much each component contributed to the error.
Formalization
For a network with L layers, you compute gradients in order: Layer L → Layer L-1 → ... → Layer 2 → Layer 1, using the recursive relationship that each layer's gradient depends on the next layer's gradient multiplied by local derivatives.
Key Takeaway: The backward pass algorithm formalizes gradient computation as a reverse-order traversal through the network, applying the chain rule layer by layer to propagate error signals from output back to input.