Bootstrap Aggregating (Bagging) Fundamentals
You'll learn: How bagging creates multiple diverse models from one dataset to reduce prediction variance and improve stability.
The Core Idea
Bootstrap Aggregating—or bagging—is an ensemble technique that trains multiple models on different random samples of your training data, then combines their predictions. The "bootstrap" part refers to creating these samples through sampling with replacement, meaning the same data point can appear multiple times in a sample.
Think of it like consulting multiple doctors who each trained on slightly different case studies. Each doctor brings a unique perspective, and their combined diagnosis is often more reliable than any single opinion.
How Bagging Works
-
Bootstrap Sampling: From your original training set of N examples, create multiple new datasets by randomly sampling N examples with replacement. Some original examples appear multiple times, others not at all—each bootstrap sample is unique.
-
Train Diverse Models: Train a separate model (like a decision tree) on each bootstrap sample. Because each sample differs, each model learns slightly different patterns.
-
Aggregate Predictions: For a new data point, get predictions from all models and combine them—typically by averaging (regression) or voting (classification).
Why It Reduces Variance
Individual decision trees are high-variance models—small changes in training data cause big changes in the tree structure. By training many trees on different samples and averaging their predictions, bagging smooths out these individual quirks. The diversity from bootstrap sampling ensures models make different errors, which cancel out when aggregated.
Key Takeaway: Bagging reduces variance by training multiple models on bootstrapped datasets (sampled with replacement) and aggregating their predictions, making ensemble predictions more stable than individual models.