Course contentsShow
← Machine Learning and Deep Learning
Lesson 574 of 3,53814. Probabilistic and Bayesian Machine LearningFree lesson

Hyperparameter Optimization via Marginal Likelihood

Tuning kernel hyperparameters by maximizing the log marginal likelihood of the observed data.

Hyperparameter Optimization via Marginal Likelihood

You'll learn: How to automatically tune Gaussian Process kernel hyperparameters by finding values that make your observed data most probable.

The Core Idea

When you build a Gaussian Process, your kernel functions have hyperparameters—like the length scale in an RBF kernel or the noise variance. But how do you choose good values? The marginal likelihood (also called "evidence") tells you how probable your observed data is under a given set of hyperparameters, after integrating out all possible functions the GP could represent.

Think of it like this: if you're fitting a curve to data points, some kernel settings will make your actual observations look natural and expected, while others will make them seem surprising or unlikely. The marginal likelihood quantifies this intuition mathematically.

Why It Works

Unlike traditional loss functions that only care about predictions, the log marginal likelihood balances two competing forces:

  • Data fit: How well the GP explains the observed data
  • Model complexity: Penalty for overly flexible models that could overfit

By maximizing the log marginal likelihood with respect to hyperparameters (using gradient-based optimizers), you find kernel settings that generalize well without needing a separate validation set.

The Process

  1. Start with initial hyperparameter guesses
  2. Compute the log marginal likelihood for your data
  3. Use optimization (gradient descent, L-BFGS) to find hyperparameters that maximize it
  4. Use these optimized hyperparameters for GP prediction

This is what happens "under the hood" when you call .fit() on a GP in most libraries—it's automatically tuning those kernel hyperparameters for you.

Key Takeaway: Marginal likelihood optimization automatically tunes GP kernel hyperparameters by finding values that make your observed data most probable, naturally balancing model fit against complexity without requiring cross-validation.