The Hypothesis Space
What you'll learn: You'll understand the set of all possible functions a machine learning model can represent, and why model complexity directly shapes what can and cannot be learned.
Core Idea
In machine learning, the hypothesis space is the complete set of functions (hypotheses) that your chosen algorithm could possibly learn from the training data. Think of it as the "solution catalog" your model can flip through when trying to find a pattern.
When you choose a model architecture—say, linear regression versus a neural network—you're actually defining the boundaries of what patterns the model can even consider. A simple linear model can only explore straight-line relationships, while a deep neural network can represent vastly more complex, non-linear patterns.
Why It Matters
The hypothesis space is constrained by:
- Model type: Linear models have simpler hypothesis spaces than decision trees or neural networks
- Parameters and capacity: More parameters mean a larger, more flexible hypothesis space
- Architecture choices: The structure you design determines what functions are reachable
The Complexity Trade-off
A small hypothesis space (simple model) may not contain a good solution if the true pattern is complex—this causes underfitting. You're searching in a phone book that doesn't have the number you need.
A large hypothesis space (complex model) contains more potential solutions, but also more bad ones. Without enough training data, the model might latch onto noise—this causes overfitting. You're searching through millions of candidates with insufficient guidance.
The art of ML is choosing a hypothesis space rich enough to contain good solutions, but constrained enough to find them reliably with your available data.
Key Takeaway: The hypothesis space defines all functions your model can learn; choosing the right model complexity balances expressiveness with the risk of overfitting or underfitting.