Course contentsShow
← Machine Learning and Deep Learning
Lesson 112 of 3,5381. Mathematical Foundations for Machine LearningFree lesson

Subgradients and Non-Smooth Optimization

Extending gradient-based methods to non-differentiable functions using subgradients.

Subgradients and Non-Smooth Optimization

What you'll learn: How to extend gradient-based optimization to functions that aren't smooth or differentiable everywhere.

The Challenge of Non-Smooth Functions

You've learned that gradient descent relies on computing gradients to find optimal solutions. But what happens when your function has sharp corners or kinks where derivatives don't exist? This is common in machine learning—think of the absolute value function |x| at x=0, or the ReLU activation function at its bend point.

What is a Subgradient?

A subgradient generalizes the concept of a gradient to non-differentiable points. At any point where a function is differentiable, the subgradient equals the regular gradient. At non-differentiable points, the subgradient is any slope of a line that stays below the function everywhere (for convex functions).

Intuitive Picture

Imagine you're at a valley with a sharp V-shaped bottom. You can't define a single downward slope at the exact bottom point, but you can define a range of slopes—any angle between the left and right sides of the V works. Each of these slopes is a valid subgradient.

Subgradient Methods

Subgradient descent works like gradient descent: pick any subgradient at your current point and take a step in its negative direction. While convergence is typically slower than standard gradient descent (and not guaranteed for every step to reduce the objective), it still finds optimal solutions for convex problems with appropriate step size rules.

This technique is essential for optimizing functions with L1 regularization, hinge loss (used in SVMs), and other non-smooth objectives common in machine learning.

Key Takeaway: Subgradients extend optimization to non-differentiable functions by replacing gradients with a set of valid slope values, enabling gradient-like methods even at sharp corners.