Convex Functions
What you'll learn: How to identify convex functions and why they're crucial for finding guaranteed optimal solutions in machine learning.
What Is a Convex Function?
A convex function is one where a line segment connecting any two points on its graph lies on or above the curve itself. Imagine a bowl sitting right-side up—that's convex. Any straight path between two points on the bowl's surface stays above (or touches) the bowl.
Mathematically, a function f is convex if for any two points x₁ and x₂ in its domain, and any value t between 0 and 1:
f(tx₁ + (1-t)x₂) ≤ tf(x₁) + (1-t)f(x₂)
This says the function's value at any weighted average of inputs is at most the weighted average of the outputs.
Testing for Convexity
For smooth functions with derivatives, you can check convexity using:
- 1D functions: The second derivative f''(x) ≥ 0 everywhere
- Multivariable functions: The Hessian matrix (from Lesson 46) is positive semidefinite everywhere
Common convex functions include f(x) = x², f(x) = eˣ, and f(x) = -log(x) on positive values.
Why Convexity Matters
Here's the magic: In a convex function, every local minimum is also a global minimum. There are no hidden valleys where optimization algorithms get trapped. When you're training a model by minimizing a convex loss function, finding any critical point with zero gradient guarantees you've found the best possible solution.
This contrasts sharply with non-convex functions (like those in deep neural networks), where local optima abound.
Key Takeaway: Convex functions have bowl-shaped graphs where any local minimum is guaranteed to be the global minimum, making optimization problems tractable and solutions reliable.