Decision Boundaries and Axis-Aligned Splits
What you'll learn: How decision trees create rectangular decision regions and why they struggle with diagonal patterns.
The Rectangular Nature of Tree Splits
Unlike logistic regression or SVMs that can create smooth, angled decision boundaries, decision trees make splits that are perfectly aligned with the feature axes. Each split asks a simple question: "Is feature X greater than value Y?" This creates vertical or horizontal cuts through your feature space, never diagonal ones.
Think of it like dividing a room using only walls that run perfectly north-south or east-west. You can never build a wall at a 45-degree angle.
Visualizing Axis-Aligned Regions
When a decision tree learns from data with two features (X₁ and X₂), every split creates a rectangular partition:
- First split: "Is X₁ < 5?" divides space with a vertical line
- Second split: "Is X₂ < 3?" divides one region with a horizontal line
- Each subsequent split further subdivides existing rectangles
The final decision boundary looks like a staircase or grid of rectangles, not a smooth curve.
The Diagonal Boundary Problem
This axis-aligned constraint creates a fundamental limitation. Suppose your data is perfectly separated by a diagonal line (like X₂ = X₁). A decision tree would need many small rectangular steps to approximate that diagonal, resulting in:
- A deeper, more complex tree
- Potential overfitting
- Less interpretable structure
Other algorithms like SVMs with kernels or logistic regression with engineered features can capture diagonal boundaries naturally with a single smooth line.
Key Takeaway: Decision trees split data using axis-aligned (horizontal/vertical) cuts only, creating rectangular decision regions that may require many splits to approximate diagonal or curved patterns that other models handle more efficiently.