Bagging: Bootstrap Aggregating
What you'll learn: How bagging creates an ensemble of diverse models by training each on a random subset of data with replacement to reduce prediction variance.
The Core Idea
Imagine you're trying to predict something important—say, housing prices. Instead of building one decision tree that might be overly sensitive to quirks in your training data, bagging builds many trees, each trained on a slightly different random sample of your data.
Here's the clever part: each sample is drawn with replacement (bootstrap sampling). This means when creating a training set for one model, you randomly pick examples from your original data, but the same example can be picked multiple times, while others might not be picked at all. Typically, each sample is the same size as your original dataset.
Why It Works
Each model sees a slightly different view of the data, so each learns slightly different patterns. Some trees might overfit to certain noise, but others won't see that same noise. When you aggregate their predictions (averaging for regression, voting for classification), the individual errors tend to cancel out.
Analogy: Instead of asking one expert who might have blind spots, you train multiple experts on slightly different textbooks, then average their opinions. Their shared insights reinforce correct patterns, while their individual mistakes get diluted.
The Variance Reduction Magic
Bagging specifically targets high-variance models (like deep decision trees that are sensitive to small data changes). By averaging many models, you keep the low bias (ability to capture complex patterns) but dramatically reduce variance (sensitivity to data fluctuations).
Key Takeaway: Bagging reduces overfitting by training multiple models on bootstrapped samples of your data and combining their predictions, trading individual model instability for collective wisdom.