The Gradient Descent Algorithm
What you'll learn: How gradient descent systematically finds optimal solutions by taking repeated steps in the direction that most reduces your objective function.
The Core Idea
Gradient descent is the workhorse algorithm for training machine learning models. Remember the gradient vector you learned earlier? It points in the direction of steepest increase of a function. Gradient descent does the opposite: it moves in the direction of steepest decrease to find a minimum.
Think of it like hiking down a foggy mountain. You can't see the bottom, but you can feel the slope beneath your feet. At each step, you move in the direction that goes downhill fastest. Eventually, you reach a valley (a local minimum).
The Iterative Update Rule
Gradient descent repeats this simple formula:
x_{new} = x_{old} - α × ∇f(x_{old})
Where:
- x is your current position (parameter values)
- α (alpha) is the learning rate — how big each step is
- ∇f(x) is the gradient — the direction of steepest ascent
- The minus sign means we go opposite the gradient (downhill)
Step-by-Step Mechanics
- Initialize: Start at some point x₀ (often randomly)
- Compute gradient: Calculate ∇f(x) at your current position
- Update: Move opposite the gradient: x = x - α × ∇f(x)
- Repeat: Go back to step 2 until convergence (gradient ≈ 0 or change becomes tiny)
The algorithm stops when you've reached a point where the gradient is nearly zero — meaning you're at (or very near) a minimum.
Key Takeaway: Gradient descent finds minima by iteratively stepping opposite to the gradient direction, using the formula x_{new} = x_{old} - α × ∇f(x) until convergence.