Recursive Feature Elimination
What you'll learn: How to systematically eliminate weak features by repeatedly training models and removing the least important ones until you find the optimal feature set.
The Core Idea
Imagine you're assembling a sports team and need to figure out which players contribute most to winning. You could start with everyone, play a season, then cut the weakest player. Repeat this process until performance starts dropping—that's when you know you've found your core team.
Recursive Feature Elimination (RFE) works exactly this way with your dataset's features. It's a wrapper method that:
- Trains a model on all features
- Ranks features by importance (using the model's built-in scoring)
- Removes the least important feature(s)
- Repeats steps 1-3 until reaching a desired number of features
Why This Matters
RFE automates the backward selection process you learned earlier, but with a smarter approach. Instead of manually testing each removal, RFE leverages model-specific importance scores (like coefficients in linear models or feature importances in trees) to guide elimination.
The "recursive" part means it continuously refines its understanding—after removing weak features, the model retrains and may discover that other features become more or less important in the new context.
The Process in Practice
Start with 20 features → train model → eliminate 2 weakest → train on 18 features → eliminate 2 more → continue until you have your target number (say, 10 features) or until performance degrades.
Key Takeaway: Recursive Feature Elimination automates feature selection by iteratively training models, ranking features by importance, and pruning the weakest ones—giving you a data-driven path to an optimal, compact feature set.