Validation Curves for Hyperparameters
What you'll learn: How to visualize model performance against a single hyperparameter to find the sweet spot that balances bias and variance.
The Core Idea
A validation curve is a diagnostic plot that shows how both training and validation scores change as you vary a single hyperparameter. While learning curves (which you just learned) show performance versus training set size, validation curves show performance versus hyperparameter values.
Think of it like adjusting the temperature on your oven: too cold and your cake doesn't cook (underfitting), too hot and it burns (overfitting). A validation curve helps you find the "just right" temperature by plotting cake quality at different settings.
How It Works
You choose one hyperparameter to investigate—for example, the maximum depth of a decision tree or the regularization strength. For each value:
- Train the model with that setting
- Record both training score and validation score
- Plot both scores against the hyperparameter values
What to Look For
- Optimal point: Where validation score peaks (or loss minimizes)
- Overfitting zone: Training score high, validation score drops—you've gone too far
- Underfitting zone: Both scores low—hyperparameter too restrictive
- Gap between curves: Shows the generalization gap
The validation curve reveals whether increasing or decreasing a hyperparameter helps, and where diminishing returns begin. This guides your grid search or random search boundaries efficiently.
Key Takeaway: Validation curves plot model performance against individual hyperparameter values, revealing the optimal setting where your model generalizes best without underfitting or overfitting.