Course contentsShow
← Machine Learning and Deep Learning
Lesson 28 of 3,5381. Mathematical Foundations for Machine LearningFree lesson

Numerical Stability in Linear Algebra

Understand conditioning, floating-point issues, and stable algorithms for matrix operations in practice.

Numerical Stability in Linear Algebra

What you'll learn: Why computers sometimes get matrix calculations "wrong" and how to avoid catastrophic errors in machine learning computations.

The Problem with Computer Arithmetic

When you compute a matrix inverse or solve a system of equations on a computer, you're not working with perfect mathematics—you're working with floating-point approximations. Computers represent numbers with finite precision (typically 64 bits), which means tiny rounding errors creep into every calculation.

What is Conditioning?

A matrix's condition number tells you how sensitive its operations are to small input errors. Think of it like a volume knob: a well-conditioned matrix (condition number near 1) is like a gentle knob—small turns give small changes. An ill-conditioned matrix is like a hyper-sensitive knob where tiny movements cause huge jumps.

For example, when computing a matrix inverse, if your matrix is nearly singular (determinant close to zero), small rounding errors in your data can produce wildly wrong results. This is a poorly conditioned problem.

Stable Algorithms in Practice

Not all mathematically equivalent methods are numerically equal:

  • Bad: Computing a matrix inverse directly, then multiplying (error-prone)
  • Good: Using QR decomposition or SVD to solve systems (more stable)
  • Bad: Normal equations for least squares: (X^T X)^(-1) X^T y
  • Good: Using SVD or QR factorization instead

When you previously learned about SVD and eigendecomposition, those weren't just theoretical tools—they're numerically stable ways to perform matrix operations that would be disastrous if done naively.

Key Takeaway: In machine learning, always prefer numerically stable algorithms (like SVD or QR decomposition) over direct inversion, especially with large or nearly-singular matrices—small rounding errors can compound into completely wrong answers.