Implementing Permutation Importance in Scikit-learn
You'll learn how to compute permutation importance using sklearn.inspection.permutation_importance with practical parameter choices for reliable results.
Core Idea
Scikit-learn provides permutation_importance() to automate the permutation procedure you've already learned conceptually. It shuffles each feature, measures performance drops, and repeats the process multiple times to produce robust importance scores with uncertainty estimates.
The Basic Implementation
from sklearn.inspection import permutation_importance
# Compute permutation importance
result = permutation_importance(
estimator=trained_model,
X=X_test, # Use test set, not training
y=y_test,
n_repeats=10, # More repeats = more stable estimates
random_state=42,
n_jobs=-1 # Parallelize across CPU cores
)
# Access results
importances_mean = result.importances_mean
importances_std = result.importances_std
Critical Parameters
scoring: Defaults to model's score method, but specify explicitly (e.g.,'neg_mean_squared_error','accuracy') for clarityn_repeats: 10-30 is typical; higher reduces noise but increases computationrandom_state: Ensures reproducibility across runsn_jobs: Use-1to leverage all CPU cores for faster computation
Example with Custom Metric
result = permutation_importance(
estimator=model,
X=X_test,
y=y_test,
scoring='roc_auc', # Specify your metric
n_repeats=20,
random_state=42
)
# Visualize with error bars
import matplotlib.pyplot as plt
sorted_idx = result.importances_mean.argsort()
plt.barh(range(len(sorted_idx)), result.importances_mean[sorted_idx])
plt.xlabel('Permutation Importance')
The function returns a Bunch object containing importances (all repetitions), importances_mean, and importances_std—giving you both point estimates and uncertainty quantification.
Key Takeaway: Use sklearn.inspection.permutation_importance() with n_repeats=10-30 on your test set to get stable, model-agnostic feature importance scores with built-in uncertainty estimates—specify your metric explicitly for clarity.