The Gradient Points in the Direction of Steepest Ascent
What you'll learn: You'll discover why the gradient vector always points in the direction where a function increases most rapidly.
The Core Idea
Imagine you're hiking on a mountainside. At any point, you can walk in infinitely many directions. Some directions lead uphill gradually, others steeply, and some lead downhill. The gradient vector always points in the direction that climbs the mountain fastest.
Mathematically, recall that the directional derivative in direction u (a unit vector) is:
This dot product equals (since u has length 1), where θ is the angle between ∇f and u.
Why the Gradient Wins
The dot product is maximized when , which happens when . This means u points in the same direction as ∇f.
In other words:
- The rate of increase in any direction is
- This rate is largest when you move parallel to ∇f
- The maximum rate of increase is exactly , the magnitude of the gradient itself
The Analogy
Think of the gradient as a compass that always points "uphill." The length of the gradient tells you how steep the climb is. If you deviate from this direction, decreases, meaning you're climbing less steeply.
Conversely, the gradient points opposite to the direction of steepest descent (the negative gradient, −∇f).
Key Takeaway: The gradient ∇f points in the direction of maximum rate of increase, and its magnitude gives that maximum rate.