Course contentsShow
← Computer Science
Lesson 2854 of 2,87263. Classical ML AlgorithmsFree lesson

Bagging: Bootstrap Aggregating

Learning how bagging creates diverse models by training on random subsets of data with replacement to reduce variance.

Bagging: Bootstrap Aggregating

What you'll learn: How bagging creates an ensemble of diverse models by training each on a random subset of data with replacement to reduce prediction variance.

The Core Idea

Imagine you're trying to predict something important—say, housing prices. Instead of building one decision tree that might be overly sensitive to quirks in your training data, bagging builds many trees, each trained on a slightly different random sample of your data.

Here's the clever part: each sample is drawn with replacement (bootstrap sampling). This means when creating a training set for one model, you randomly pick examples from your original data, but the same example can be picked multiple times, while others might not be picked at all. Typically, each sample is the same size as your original dataset.

Why It Works

Each model sees a slightly different view of the data, so each learns slightly different patterns. Some trees might overfit to certain noise, but others won't see that same noise. When you aggregate their predictions (averaging for regression, voting for classification), the individual errors tend to cancel out.

Analogy: Instead of asking one expert who might have blind spots, you train multiple experts on slightly different textbooks, then average their opinions. Their shared insights reinforce correct patterns, while their individual mistakes get diluted.

The Variance Reduction Magic

Bagging specifically targets high-variance models (like deep decision trees that are sensitive to small data changes). By averaging many models, you keep the low bias (ability to capture complex patterns) but dramatically reduce variance (sensitivity to data fluctuations).

Key Takeaway: Bagging reduces overfitting by training multiple models on bootstrapped samples of your data and combining their predictions, trading individual model instability for collective wisdom.