Course contentsShow
← Machine Learning and Deep Learning
Lesson 3204 of 3,53870. Interpretability: Feature-Based MethodsFree lesson

Implementing Permutation Importance in Scikit-learn

Using sklearn.inspection.permutation_importance with practical examples and parameter choices for robust results.

Implementing Permutation Importance in Scikit-learn

You'll learn how to compute permutation importance using sklearn.inspection.permutation_importance with practical parameter choices for reliable results.

Core Idea

Scikit-learn provides permutation_importance() to automate the permutation procedure you've already learned conceptually. It shuffles each feature, measures performance drops, and repeats the process multiple times to produce robust importance scores with uncertainty estimates.

The Basic Implementation

from sklearn.inspection import permutation_importance

# Compute permutation importance
result = permutation_importance(
    estimator=trained_model,
    X=X_test,  # Use test set, not training
    y=y_test,
    n_repeats=10,  # More repeats = more stable estimates
    random_state=42,
    n_jobs=-1  # Parallelize across CPU cores
)

# Access results
importances_mean = result.importances_mean
importances_std = result.importances_std

Critical Parameters

  • scoring: Defaults to model's score method, but specify explicitly (e.g., 'neg_mean_squared_error', 'accuracy') for clarity
  • n_repeats: 10-30 is typical; higher reduces noise but increases computation
  • random_state: Ensures reproducibility across runs
  • n_jobs: Use -1 to leverage all CPU cores for faster computation

Example with Custom Metric

result = permutation_importance(
    estimator=model,
    X=X_test,
    y=y_test,
    scoring='roc_auc',  # Specify your metric
    n_repeats=20,
    random_state=42
)

# Visualize with error bars
import matplotlib.pyplot as plt
sorted_idx = result.importances_mean.argsort()
plt.barh(range(len(sorted_idx)), result.importances_mean[sorted_idx])
plt.xlabel('Permutation Importance')

The function returns a Bunch object containing importances (all repetitions), importances_mean, and importances_std—giving you both point estimates and uncertainty quantification.

Key Takeaway: Use sklearn.inspection.permutation_importance() with n_repeats=10-30 on your test set to get stable, model-agnostic feature importance scores with built-in uncertainty estimates—specify your metric explicitly for clarity.