Course contentsShow
← Machine Learning and Deep Learning
Lesson 218 of 3,5383. Linear RegressionFree lesson

Convergence Criteria and Stopping Conditions

Define practical stopping rules based on gradient magnitude, cost change, and maximum iteration limits.

Convergence Criteria and Stopping Conditions

You'll learn: How to decide when gradient descent has done "enough" work and should stop training.

Why We Need Stopping Rules

When running gradient descent (batch, stochastic, or mini-batch), the algorithm updates parameters repeatedly. But how do you know when to stop? You can't run it forever! You need clear, practical rules to decide when your model has converged or when continuing would be wasteful.

Think of it like baking bread: you don't leave it in the oven indefinitely. You check if it's golden brown (visual criterion), if the timer went off (time limit), or if the temperature stabilized (change criterion). Similarly, gradient descent needs multiple ways to know when it's "done."

Three Essential Stopping Criteria

1. Gradient Magnitude

When the gradient becomes very small (close to zero), you're near a minimum. If the gradient's magnitude falls below a threshold like 1e-6, stop—you've essentially "flattened out."

2. Cost Change

Monitor how much the cost function (MSE) decreases between iterations. If the improvement becomes tiny (e.g., less than 1e-4), you're getting diminishing returns. Stop and save computation.

3. Maximum Iterations

Set a hard limit (like 1000 or 10,000 iterations) as a safety net. This prevents infinite loops if convergence is slow or if your learning rate is poorly chosen.

Combining Criteria

In practice, use all three: stop when any condition is met. This makes your training robust—you're protected against both premature stopping and endless computation.

Key Takeaway: Convergence criteria (gradient magnitude, cost change, max iterations) tell gradient descent when to stop, balancing computational efficiency with model quality.