Matrix Calculus: Gradients of Matrix Expressions
What You'll Learn
You'll learn how to compute derivatives when your function outputs a single number (scalar) but depends on entire vectors or matrices—a critical skill for understanding how machine learning models learn.
Why This Matters
In machine learning, we constantly adjust parameters (stored as vectors or matrices) to minimize error functions (scalars like "total loss"). To know which direction to adjust these parameters, we need derivatives—but now we're taking derivatives with respect to entire collections of numbers at once.
The Core Ideas
Gradient with Respect to a Vector
When you have a scalar function f that depends on a vector x = [x₁, x₂, ..., xₙ], the gradient ∇f is a vector containing all the partial derivatives:
∇f = [∂f/∂x₁, ∂f/∂x₂, ..., ∂f/∂xₙ]
Think of it as asking: "If I nudge each component of x slightly, how much does f change?"
Gradient with Respect to a Matrix
Similarly, if f depends on a matrix W, the gradient ∇f is a matrix of the same shape, where each entry is ∂f/∂Wᵢⱼ.
Common Example: Quadratic Forms
Remember quadratic forms like f(x) = xᵀA****x? The gradient is: ∇f = (A + Aᵀ)x
If A is symmetric (which it often is), this simplifies to 2A****x.
Analogy
Imagine hiking on terrain where your position is a vector (latitude, longitude, elevation). The gradient tells you the direction of steepest ascent at your current position—it's a vector pointing uphill.
Key Takeaway: The gradient of a scalar function with respect to a vector or matrix is a same-shaped object containing all partial derivatives, pointing in the direction of greatest increase.