Hyperparameter Optimization via Marginal Likelihood
You'll learn: How to automatically tune Gaussian Process kernel hyperparameters by finding values that make your observed data most probable.
The Core Idea
When you build a Gaussian Process, your kernel functions have hyperparameters—like the length scale in an RBF kernel or the noise variance. But how do you choose good values? The marginal likelihood (also called "evidence") tells you how probable your observed data is under a given set of hyperparameters, after integrating out all possible functions the GP could represent.
Think of it like this: if you're fitting a curve to data points, some kernel settings will make your actual observations look natural and expected, while others will make them seem surprising or unlikely. The marginal likelihood quantifies this intuition mathematically.
Why It Works
Unlike traditional loss functions that only care about predictions, the log marginal likelihood balances two competing forces:
- Data fit: How well the GP explains the observed data
- Model complexity: Penalty for overly flexible models that could overfit
By maximizing the log marginal likelihood with respect to hyperparameters (using gradient-based optimizers), you find kernel settings that generalize well without needing a separate validation set.
The Process
- Start with initial hyperparameter guesses
- Compute the log marginal likelihood for your data
- Use optimization (gradient descent, L-BFGS) to find hyperparameters that maximize it
- Use these optimized hyperparameters for GP prediction
This is what happens "under the hood" when you call .fit() on a GP in most libraries—it's automatically tuning those kernel hyperparameters for you.
Key Takeaway: Marginal likelihood optimization automatically tunes GP kernel hyperparameters by finding values that make your observed data most probable, naturally balancing model fit against complexity without requiring cross-validation.