The Chain Rule
What you'll learn: How to find derivatives of nested functions—the mathematical backbone of training neural networks.
What Is the Chain Rule?
When you have a composite function (a function inside another function), the chain rule tells you how to compute its derivative by breaking it down into steps. Think of it like peeling an onion: you work from the outside layer inward, multiplying the derivatives as you go.
In mathematical notation: if you have a composite function y = f(g(x)), the chain rule states:
dy/dx = (df/dg) × (dg/dx)
Or more intuitively: the derivative of the outer function (evaluated at the inner function) times the derivative of the inner function.
Why It Matters in Machine Learning
Neural networks are massive compositions of functions. When you train them, you need to compute how much each parameter affects the final error—that's where the chain rule comes in. Every layer in a network transforms data through a function, and these transformations are nested inside each other. To update weights, you propagate derivatives backward through all these layers using... the chain rule!
A Simple Example
Suppose y = (3x + 2)². This is f(g(x)) where:
- Inner function:
g(x) = 3x + 2 - Outer function:
f(g) = g²
Apply the chain rule:
- Derivative of outer function:
df/dg = 2g - Derivative of inner function:
dg/dx = 3 - Multiply them:
dy/dx = 2g × 3 = 2(3x + 2) × 3 = 6(3x + 2)
At x = 1: dy/dx = 6(5) = 30
Key Takeaway: The chain rule decomposes complex derivatives into manageable pieces—multiply the derivative of each layer as you work inward. This principle powers backpropagation in neural networks.