Course contentsShow
← Machine Learning and Deep Learning
Lesson 510 of 3,53812. Model Validation and TuningFree lesson

Bayesian Optimization Fundamentals

Model the objective function with a probabilistic surrogate to guide intelligent search.

Bayesian Optimization Fundamentals

What you'll learn: How Bayesian optimization uses a probabilistic model to intelligently decide which hyperparameters to test next, making search far more efficient than random or grid approaches.

The Core Problem

Grid search and random search treat every hyperparameter combination equally—they're "blind" explorers. Grid search checks everything systematically, while random search samples blindly. Both waste time testing obviously poor configurations. What if we could learn from previous trials and make smarter guesses about where good hyperparameters might be?

How Bayesian Optimization Works

Bayesian optimization builds a probabilistic surrogate model (usually a Gaussian Process) that approximates your objective function—say, validation accuracy as a function of learning rate and regularization strength.

Think of it like a treasure map that gets clearer with each dig. Initially, the map (surrogate model) has high uncertainty everywhere. You pick a promising spot based on balancing:

  • Exploitation: digging where the map suggests treasure is likely
  • Exploration: checking uncertain areas that might hide even better treasure

After each trial, you update the map with the result. The surrogate model becomes more confident about that region and guides your next choice. This cycle repeats, efficiently zeroing in on optimal hyperparameters.

The Advantage

Unlike grid or random search that might waste hundreds of evaluations, Bayesian optimization typically finds excellent hyperparameters in 20-100 trials by intelligently learning which regions of the search space are promising and which to avoid.

Key Takeaway: Bayesian optimization uses a probabilistic surrogate model to learn from past hyperparameter trials, intelligently balancing exploration of uncertain regions with exploitation of known good areas, making it far more sample-efficient than blind search methods.