Handling Categorical Features in Trees
What you'll learn: How decision trees naturally work with categorical data and why splitting on categories can be surprisingly complex.
Why Trees Like Categories
Unlike many machine learning algorithms that require encoding categorical features into numbers, decision trees can work with categorical variables quite naturally. This is because trees make decisions by asking yes/no questions, and "Is city = 'Paris'?" is just as valid as "Is temperature > 20?"
The Simplicity of Binary Categorical Features
For a binary categorical variable (like "Has_Degree: Yes/No"), the tree simply creates a split that separates the two values. Easy!
The Challenge of Multiway Splits
Here's where it gets interesting. Imagine you have a "Color" feature with values: Red, Blue, Green, Yellow.
A decision tree could handle this in two ways:
- Multiway split: Create 4 branches at once (one per color)
- Binary split: Ask "Is Color = Red?" then recurse on remaining colors
The complexity problem: With k categories, there are 2^(k-1) - 1 possible ways to group them into two sets. For just 10 categories, that's over 500 possible binary splits to evaluate!
Most modern implementations (including scikit-learn) convert categorical features to integers and treat them as ordered, which isn't ideal but is computationally manageable. Some specialized implementations (like LightGBM) use optimized algorithms to find good categorical splits efficiently.
The Ordering Issue
When trees treat "Red=1, Blue=2, Green=3" as ordered numbers, they might split at "≤ 2", grouping Red and Blue together. This grouping might be meaningless unless there's a natural order to your categories.
Key Takeaway: Decision trees handle categorical features naturally in principle, but multiway splits create exponential complexity, so most implementations use approximations or require manual encoding strategies like one-hot encoding.