Adjusted R² for Model Comparison
What you'll learn: How adjusted R² corrects for model complexity to fairly compare regression models with different numbers of features.
The Problem with R²
You've learned that R² score measures how well your model explains variance in the target variable. But R² has a quirk: it always increases (or stays the same) when you add more features—even if those features are completely useless! This makes it unreliable when comparing models with different numbers of predictors.
What Adjusted R² Does
Adjusted R² applies a penalty for each additional feature you add to your model. Think of it like this: if you're judging two students' essays, one who wrote 500 words and another who wrote 2000 words, adjusted R² is like grading "quality per word" instead of just total quality. The longer essay doesn't automatically win.
The formula adjusts R² based on:
- Number of features (p): More features = bigger penalty
- Sample size (n): Larger datasets reduce the penalty per feature
Why It Matters for Comparison
Imagine comparing two models:
- Model A: 3 features, R² = 0.85
- Model B: 20 features, R² = 0.87
Model B looks better, but is it really? Adjusted R² might reveal that Model A is actually superior because Model B's slight improvement doesn't justify adding 17 extra features. The simpler model is more efficient.
When to Use It
Use adjusted R² when:
- Comparing models with different numbers of features
- Deciding whether to keep or remove predictors
- Avoiding overfitting by penalizing unnecessary complexity
Key Takeaway: Adjusted R² penalizes model complexity, making it the right metric for comparing regression models with different feature counts—it rewards simplicity and punishes bloat.