Course contentsShow
← Machine Learning and Deep Learning
Lesson 25 of 3,5381. Mathematical Foundations for Machine LearningFree lesson

Positive Definite and Semidefinite Matrices

Learn definitions, properties, and why positive definiteness matters for optimization in ML.

Positive Definite and Semidefinite Matrices

You'll learn what makes a matrix "positive definite" and why this property is crucial for optimization algorithms in machine learning.

What Are These Matrices?

A symmetric matrix A is called:

  • Positive definite if for any non-zero vector x, the quantity xᵀAx is always positive (> 0)
  • Positive semidefinite if xᵀAx is always non-negative (≥ 0)

Think of xᵀAx as a "weighted score." A positive definite matrix guarantees this score is always positive unless x is the zero vector.

Properties That Matter

From eigenvalues and eigenvectors (concepts you've learned), here's a key insight:

  • A matrix is positive definite when all its eigenvalues are positive
  • A matrix is positive semidefinite when all eigenvalues are non-negative

For example, the identity matrix is positive definite because all its eigenvalues equal 1.

Why This Matters for ML

Positive definite matrices appear everywhere in optimization:

  1. Convexity: When the Hessian matrix (second derivatives) of a function is positive definite, you have a unique minimum—optimization algorithms can confidently find it
  2. Covariance matrices in statistics are always positive semidefinite, ensuring distances and probabilities make sense
  3. Numerical stability: Positive definite matrices are invertible and well-behaved computationally

Analogy: Imagine a bowl (positive definite) versus a flat surface with valleys (semidefinite). In the bowl, there's one clear bottom; optimization easily finds it. The flat surface has ambiguous "bottoms."

Key Takeaway: Positive definite matrices guarantee well-behaved optimization landscapes with unique minima, making them essential for training machine learning models reliably.