Course contentsShow
← Machine Learning and Deep Learning
Lesson 26 of 3,5381. Mathematical Foundations for Machine LearningFree lesson

Quadratic Forms

Understand quadratic forms x^TAx and their connection to loss functions and optimization landscapes.

Quadratic Forms

What you'll learn: How a special mathematical expression called a quadratic form connects to the shapes of loss surfaces you'll encounter in machine learning optimization.

What Is a Quadratic Form?

A quadratic form is an expression that looks like x^T A x, where:

  • x is a vector (like your model parameters)
  • A is a square matrix (usually symmetric)
  • The result is a single scalar value

Think of it as a way to map any vector to a number using a matrix "in the middle."

Why Does This Matter?

When you train a machine learning model, you're trying to minimize a loss function. Near any point, many loss functions can be approximated by a quadratic form. The matrix A encodes the curvature of your loss landscape:

  • If A is positive definite (from the previous lesson), the quadratic form creates a bowl shape—a nice valley where optimization algorithms can roll downhill to find the minimum
  • If A has negative eigenvalues, you get saddle points—tricky spots where some directions go up and others go down
  • The eigenvalues of A tell you how steep the bowl is in different directions

Simple Analogy

Imagine standing in a landscape. A quadratic form is like a mathematical description of the terrain's shape around you. Matrix A tells you: "Is this a valley? A hilltop? How steep are the sides?" Understanding this helps optimization algorithms decide which direction to step.

Key Takeaway: Quadratic forms x^T A x describe the local shape of loss functions, and their matrix A reveals whether you're in a nice optimization valley (positive definite) or a challenging saddle region.