One-Hot Encoding Overview
What you'll learn: You'll discover when and how to create a dummy variable for every category—and when this approach makes sense versus using a reference category.
The Core Idea
You already know about dummy variable encoding where you create k-1 dummy variables for k categories, leaving one as the reference. One-hot encoding takes a different approach: it creates k dummy variables for k categories—one for every level, with no reference category left out.
For example, if you have a color variable with values "Red," "Green," and "Blue," one-hot encoding creates three separate binary columns: color_Red, color_Green, and color_Blue. Each observation gets a 1 in exactly one column and 0s elsewhere.
When Is One-Hot Encoding Appropriate?
One-hot encoding is useful in these scenarios:
- Tree-based models (decision trees, random forests): These algorithms don't use the same linear framework as regression and can handle all k variables without issues
- Neural networks: Many deep learning frameworks expect one-hot encoded inputs
- When interpretability requires seeing all categories: Sometimes you want explicit coefficients for every group
- Non-regression contexts: Classification tasks, clustering, or similarity calculations
The Critical Caveat
In linear regression, one-hot encoding creates perfect multicollinearity. If you include all k dummy variables plus an intercept, the model cannot uniquely estimate coefficients (the columns sum to a constant). This is why standard regression uses k-1 dummy variables with a reference category.
Key Takeaway: One-hot encoding creates a dummy variable for every category level; use it for tree-based models and neural networks, but stick with k-1 dummy encoding (with a reference category) for linear regression to avoid multicollinearity.