Advantages and Limitations of Decision Trees
You'll learn when decision trees shine and where they struggle, helping you choose the right tool for your problem.
When Decision Trees Excel
Decision trees offer several compelling advantages:
Interpretability: Trees mirror human decision-making. You can follow the exact path from root to leaf, explaining why a prediction was made—crucial in healthcare, finance, or legal applications where transparency matters.
No Feature Scaling Required: Unlike SVMs or logistic regression, trees don't care if one feature ranges from 0-1 and another from 0-10,000. They split on thresholds, making the actual scale irrelevant.
Handle Mixed Data Types: Trees naturally work with both numerical and categorical features without special encoding (though implementation details vary).
Capture Non-Linearity: Despite making axis-aligned splits, trees can approximate complex, non-linear relationships by creating enough splits—no need for polynomial features or kernels.
Automatic Feature Selection: Irrelevant features simply won't appear in important splits.
Where Decision Trees Struggle
However, trees have significant limitations:
Instability: Small changes in training data can produce completely different trees. If one patient's age shifts slightly, the entire tree structure might reorganize—this high variance makes trees unreliable.
Overfitting: Without constraints (max depth, min samples), trees memorize training data by creating leaves for individual samples. They achieve perfect training accuracy but fail on new data.
Axis-Aligned Splits Only: Trees can't create diagonal boundaries. To separate two classes along a 45° line requires many jagged, staircase-like splits—inefficient compared to linear models.
Biased Toward Dominant Classes: In imbalanced datasets, trees favor the majority class when calculating impurity.
Key Takeaway: Decision trees excel at interpretability and require minimal preprocessing, but their instability and tendency to overfit often make ensemble methods (which we'll explore next) more practical for production systems.