Course contentsShow
← Machine Learning and Deep Learning
Lesson 109 of 3,5381. Mathematical Foundations for Machine LearningFree lesson

Coordinate Descent

Optimizing one variable at a time while fixing others; useful for structured and sparse problems.

Coordinate Descent

What you'll learn: How to optimize complex problems by tackling one variable at a time instead of all at once.

The Core Idea

Imagine you're trying to find the lowest point in a hilly valley, but you're only allowed to move in perfectly straight lines—either north-south or east-west, never diagonally. That's essentially coordinate descent! Instead of computing the gradient with respect to all variables simultaneously (like in gradient descent), coordinate descent optimizes one variable at a time while keeping all others fixed.

Here's how it works: if you have a function f(x₁, x₂, ..., xₙ) you want to minimize, coordinate descent cycles through the variables. First, minimize with respect to x₁ while treating x₂, ..., xₙ as constants. Then minimize with respect to x₂ while fixing the others, and so on. Repeat this cycle until convergence.

Why Use Coordinate Descent?

This approach shines when:

  • Sparse problems: Many machine learning problems have sparse solutions (most coefficients are zero), and coordinate descent can efficiently identify and update only the relevant variables
  • Structured problems: When optimizing each individual variable is computationally cheap or has a closed-form solution
  • High dimensions: Sometimes optimizing one coordinate at a time is simpler than computing the full gradient

The method is particularly popular in LASSO regression and other L1-regularized problems where updating individual coordinates can be done very efficiently.

Connection to Previous Methods

Unlike gradient descent (which moves in the direction of steepest descent) or Newton's method (which uses second-order information), coordinate descent exploits problem structure by decomposing a complex optimization into simpler one-dimensional problems.

Key Takeaway: Coordinate descent optimizes one variable at a time while fixing others—a "divide and conquer" strategy that's highly effective for sparse and structured machine learning problems.