Course contentsShow
← Machine Learning and Deep Learning
Lesson 107 of 3,5381. Mathematical Foundations for Machine LearningFree lesson

Newton's Method

Second-order optimization using the Hessian for faster convergence; computational trade-offs in high dimensions.

Newton's Method

What You'll Learn

You'll discover how Newton's Method uses second-order information (the Hessian) to accelerate optimization, and why it's powerful but computationally expensive in high dimensions.

The Core Idea

Remember gradient descent? It only uses first-order information (the gradient) to decide which direction to step. Newton's Method goes further—it uses both the gradient and the Hessian matrix (second derivatives) to make smarter steps.

Think of it like this: gradient descent is like walking downhill in fog, feeling only the slope beneath your feet. Newton's Method is like having a detailed topographic map that shows not just the slope, but also how the slope is curving. This curvature information lets you take better shortcuts toward the minimum.

How It Works

The Newton's Method update rule is:

x_{new} = x_{old} - [H(x)]^(-1) ∇f(x)

Where H(x) is the Hessian matrix you learned about earlier. By multiplying the gradient by the inverse Hessian, you're essentially rescaling your step to account for the curvature of the function.

The Trade-off

Why it's powerful: Newton's Method typically converges much faster than gradient descent—often in just a few iterations for well-behaved functions. It's quadratically convergent near the optimum.

Why it's costly: Computing the Hessian requires storing an n×n matrix (where n is the number of parameters), and inverting it costs O(n³) operations. In deep learning with millions of parameters, this becomes prohibitively expensive.

Key Takeaway: Newton's Method offers faster convergence by using second-order (Hessian) information, but the computational cost of computing and inverting the Hessian makes it impractical for high-dimensional problems like modern deep learning.