Course contentsShow
← Machine Learning and Deep Learning
Lesson 316 of 3,5386. Tree-Based MethodsFree lesson

XGBoost Hyperparameters and Tuning

Tuning tree depth, learning rate, subsample ratios, and regularization parameters for optimal XGBoost performance.

XGBoost Hyperparameters and Tuning

What you'll learn: How to tune XGBoost's key hyperparameters to balance model complexity, training speed, and predictive performance.

Why Hyperparameter Tuning Matters

XGBoost is powerful, but it has many knobs to turn. Without proper tuning, you might get a model that overfits badly or trains too slowly. Think of hyperparameters as the recipe adjustments for baking: too much of one ingredient ruins the cake, but the right balance creates something excellent.

Core XGBoost Hyperparameters

Tree Structure Parameters

  • max_depth: Controls how deep each tree grows (typically 3-10). Deeper trees capture more complex patterns but risk overfitting.
  • min_child_weight: Minimum sum of instance weights needed in a leaf. Higher values prevent learning overly specific patterns.

Learning Control

  • learning_rate (eta): Shrinks each tree's contribution (typically 0.01-0.3). Lower rates require more trees but often generalize better—this is the shrinkage concept you learned earlier.
  • n_estimators: Number of boosting rounds (trees to build). More trees can improve performance until diminishing returns or overfitting.

Randomness and Sampling

  • subsample: Fraction of training samples used per tree (0.5-1.0). Acts like bagging to reduce overfitting.
  • colsample_bytree: Fraction of features used per tree. Similar to Random Forest's feature randomness.

Regularization

  • reg_alpha (L1) and reg_lambda (L2): Add penalties to leaf weights, preventing overly large weights that signal overfitting.

Tuning Strategy

Start with moderate values, then adjust systematically: tune tree structure first, then learning rate with more estimators, finally add regularization if needed. Use cross-validation to measure true performance.

Key Takeaway: XGBoost hyperparameters control tree complexity (depth, weights), learning speed (rate, estimators), randomness (subsample ratios), and regularization—tuning them systematically prevents overfitting while maximizing predictive power.