Course contentsShow
← Machine Learning and Deep Learning
Lesson 27 of 3,5381. Mathematical Foundations for Machine LearningFree lesson

Matrix Calculus: Gradients of Matrix Expressions

Learn to compute derivatives of scalar functions with respect to vectors and matrices.

Matrix Calculus: Gradients of Matrix Expressions

What You'll Learn

You'll learn how to compute derivatives when your function outputs a single number (scalar) but depends on entire vectors or matrices—a critical skill for understanding how machine learning models learn.

Why This Matters

In machine learning, we constantly adjust parameters (stored as vectors or matrices) to minimize error functions (scalars like "total loss"). To know which direction to adjust these parameters, we need derivatives—but now we're taking derivatives with respect to entire collections of numbers at once.

The Core Ideas

Gradient with Respect to a Vector

When you have a scalar function f that depends on a vector x = [x₁, x₂, ..., xₙ], the gradient ∇f is a vector containing all the partial derivatives:

∇f = [∂f/∂x₁, ∂f/∂x₂, ..., ∂f/∂xₙ]

Think of it as asking: "If I nudge each component of x slightly, how much does f change?"

Gradient with Respect to a Matrix

Similarly, if f depends on a matrix W, the gradient ∇f is a matrix of the same shape, where each entry is ∂f/∂Wᵢⱼ.

Common Example: Quadratic Forms

Remember quadratic forms like f(x) = xᵀA****x? The gradient is: ∇f = (A + Aᵀ)x

If A is symmetric (which it often is), this simplifies to 2A****x.

Analogy

Imagine hiking on terrain where your position is a vector (latitude, longitude, elevation). The gradient tells you the direction of steepest ascent at your current position—it's a vector pointing uphill.

Key Takeaway: The gradient of a scalar function with respect to a vector or matrix is a same-shaped object containing all partial derivatives, pointing in the direction of greatest increase.