The Instance-Based Learning Paradigm
What you'll learn: How instance-based learning differs from model-based approaches by deferring all learning until prediction time.
A Different Philosophy
So far, you've learned algorithms like logistic regression, SVMs, and decision trees that build explicit models during training. They analyze the data, extract patterns, compute parameters or split rules, and create a compact model. Once trained, they often discard the training data itself.
Instance-based learning takes the opposite approach: it stores all (or most) training examples and performs no generalization until you ask for a prediction. Think of it like open-book learning versus memorizing. Instead of studying the material and creating summary notes (a model), you keep the entire textbook and look up answers when needed.
How It Works
When training an instance-based learner:
- Store the training examples in memory
- That's it—no weights, no tree structures, no equations
When making a prediction:
- Compare the new example to stored training examples
- Find the "nearest neighbors" based on similarity
- Make a decision based on those neighbors' labels
This is called lazy learning because the algorithm postpones all computational work until prediction time, unlike eager learning methods (like SVMs or Random Forests) that do heavy computation during training.
Why This Matters
Instance-based methods like K-Nearest Neighbors (KNN) can naturally capture complex, local patterns without assuming any global structure. They adapt to the local neighborhood of each prediction, making them flexible for irregular decision boundaries.
The tradeoff? Prediction can be slow (must compare against all stored examples), and memory usage grows with your dataset size.
Key Takeaway: Instance-based learning stores training examples and defers all learning until prediction time, making decisions based on local neighborhoods rather than global model parameters.