Quasi-Newton Methods
What you'll learn: How to approximate the expensive Hessian matrix to get fast convergence without the heavy computational cost of Newton's method.
The Core Problem
You've learned that Newton's method uses both the gradient (first derivative) and the Hessian matrix (second derivatives) to find optimal solutions quickly. The problem? Computing and inverting the Hessian is extremely expensive—especially when you have thousands or millions of parameters in machine learning models.
The Quasi-Newton Solution
Quasi-Newton methods are like taking Newton's method and saying, "What if we approximated the Hessian instead of computing it exactly?"
Think of it like navigation: Newton's method is like recalculating your entire route from scratch at every intersection using satellite data (accurate but slow). Quasi-Newton methods are like updating your mental map based on the turns you've already made (fast and still pretty accurate).
The most popular quasi-Newton method is BFGS (named after Broyden, Fletcher, Goldfarb, and Shanno). It builds an approximation of the Hessian by observing how the gradient changes as you move through parameter space. Each step, it updates this approximation using only gradient information—no expensive second derivative calculations needed.
Why It Matters
Quasi-Newton methods strike a sweet spot:
- Faster than gradient descent: They use curvature information (like Newton's method) to take smarter steps
- Cheaper than Newton's method: No need to compute or invert the full Hessian matrix
- Practical for medium-sized problems: Common in traditional ML optimization before deep learning scaled up to billions of parameters
For problems with hundreds to thousands of parameters, BFGS often converges in far fewer iterations than gradient descent.
Key Takeaway: Quasi-Newton methods approximate the Hessian matrix using only gradient information, delivering Newton-like convergence speed at a fraction of the computational cost.