Convergence Criteria and Stopping Conditions
You'll learn: How to decide when gradient descent has done "enough" work and should stop training.
Why We Need Stopping Rules
When running gradient descent (batch, stochastic, or mini-batch), the algorithm updates parameters repeatedly. But how do you know when to stop? You can't run it forever! You need clear, practical rules to decide when your model has converged or when continuing would be wasteful.
Think of it like baking bread: you don't leave it in the oven indefinitely. You check if it's golden brown (visual criterion), if the timer went off (time limit), or if the temperature stabilized (change criterion). Similarly, gradient descent needs multiple ways to know when it's "done."
Three Essential Stopping Criteria
1. Gradient Magnitude
When the gradient becomes very small (close to zero), you're near a minimum. If the gradient's magnitude falls below a threshold like 1e-6, stop—you've essentially "flattened out."
2. Cost Change
Monitor how much the cost function (MSE) decreases between iterations. If the improvement becomes tiny (e.g., less than 1e-4), you're getting diminishing returns. Stop and save computation.
3. Maximum Iterations
Set a hard limit (like 1000 or 10,000 iterations) as a safety net. This prevents infinite loops if convergence is slow or if your learning rate is poorly chosen.
Combining Criteria
In practice, use all three: stop when any condition is met. This makes your training robust—you're protected against both premature stopping and endless computation.
Key Takeaway: Convergence criteria (gradient magnitude, cost change, max iterations) tell gradient descent when to stop, balancing computational efficiency with model quality.