Feature Selection: Filter Methods
What you'll learn: How to use simple statistical tests and correlation metrics to quickly identify which features have the strongest individual relationships with your target variable.
The Core Idea
Imagine you're a detective trying to solve a crime, and you have 100 potential clues. Instead of investigating all of them deeply (which takes time), you first quickly scan each clue individually to see which ones seem most relevant to the case. Filter methods work the same way—they rapidly evaluate each feature on its own merits before any machine learning happens.
Filter methods are statistical preprocessing techniques that rank features based on their univariate (one-at-a-time) relationship with the target variable. They're called "filter" methods because they filter out weak features early, independent of any specific model.
Common Filter Techniques
For numerical features and targets:
- Correlation coefficients (Pearson, Spearman): Measure linear or monotonic relationships between feature and target
- Mutual information: Captures any kind of relationship, including nonlinear ones
For categorical features or classification targets:
- Chi-squared test: Examines independence between categorical variables
- ANOVA F-statistic: Tests if feature means differ significantly across target classes
These methods give each feature a score. You then select the top-K features or those exceeding a threshold.
Why Use Filter Methods?
They're fast and model-agnostic—you can run them once and apply the results to any algorithm. They work well when you have hundreds or thousands of features and need a quick first pass to eliminate obviously irrelevant ones.
Key Takeaway: Filter methods use statistical tests to score each feature individually based on its relationship with the target, providing a fast, model-independent way to identify potentially useful features.