Course contentsShow
← Machine Learning and Deep Learning
Lesson 445 of 3,53810. Feature EngineeringFree lesson

Wrapper Methods: Forward and Backward Selection

Iteratively add or remove features while evaluating model performance to find optimal feature subsets.

Wrapper Methods: Forward and Backward Selection

What You'll Learn

You'll learn how to systematically build or prune feature sets by iteratively testing each feature's impact on model performance.

The Core Idea

Unlike filter methods that evaluate features independently, wrapper methods actually train models to decide which features to keep. They "wrap" around your model, testing different feature combinations to find the best subset.

Think of it like assembling a sports team: you could read player stats alone (filter methods), or you could actually watch players perform together in scrimmage games (wrapper methods). The second approach is slower but accounts for how players work together.

Forward Selection

Start with zero features and add them one at a time:

  1. Train models with each remaining feature individually
  2. Add the feature that improves performance most
  3. Repeat until performance stops improving or you reach a feature limit

It's like building a recipe by adding one ingredient at a time, tasting after each addition.

Backward Selection (Backward Elimination)

Start with all features and remove them one at a time:

  1. Train a model with all features
  2. Remove each feature one-by-one and measure performance drop
  3. Permanently remove the feature whose absence hurts performance least
  4. Repeat until performance degrades unacceptably

This is like decluttering: you remove items and see if you miss them.

Why Use Wrapper Methods?

  • Captures interactions: Sees how features work together, not just individually
  • Model-specific: Finds features optimal for your specific model
  • More accurate than filter methods (but much slower)

The tradeoff? Computational cost. Training hundreds of models takes time, especially with large datasets.

Key Takeaway: Wrapper methods iteratively add (forward selection) or remove (backward selection) features while training actual models, finding feature subsets that optimize performance for your specific model—at the cost of significant computation time.