Course contentsShow
← Machine Learning and Deep Learning
Lesson 3186 of 3,53870. Interpretability: Feature-Based MethodsFree lesson

Feature Importance: Core Concept

Understand what feature importance means, how it quantifies each feature's contribution, and limitations of different definitions.

Feature Importance: Core Concept

What you'll learn: How to quantify and understand which input features drive your model's predictions.

What Is Feature Importance?

Feature importance measures how much each input feature contributes to your model's predictions. Think of it like determining which ingredients in a recipe have the biggest impact on taste—some features strongly influence outcomes, while others barely matter.

At its core, feature importance answers: "If I changed this feature, how much would my predictions change?" This helps you understand what your model has learned and whether it's focusing on meaningful patterns or spurious correlations.

Why Multiple Definitions Exist

There's no single "correct" way to measure importance because different perspectives matter:

  • Permutation importance: Measures performance drop when you shuffle a feature's values
  • Weight-based importance: Uses model coefficients or attention scores
  • Gain-based importance: Tracks how much a feature reduces prediction error (common in tree models)
  • Drop-column importance: Compares performance with vs without each feature

Each definition captures something different. A feature might have high weight but low permutation importance if other correlated features compensate when it's removed.

Key Limitations

Correlation confounds importance. If two features are highly correlated, importance might be split between them arbitrarily, or concentrated in whichever the model happened to use first.

Context matters. A feature might be globally unimportant but crucial for specific slices of data. Aggregate importance scores hide these nuances.

Causation isn't implied. High importance doesn't mean the feature causes the outcome—only that it's predictive in your training data.

Key Takeaway: Feature importance quantifies each feature's contribution to predictions, but different definitions measure different aspects, and correlation, context-dependence, and causal interpretation require careful consideration.