Course contentsShow
← Machine Learning and Deep Learning
Lesson 564 of 3,53814. Probabilistic and Bayesian Machine LearningFree lesson

Hyperparameters and Evidence Approximation

Tune precision parameters for prior and noise by maximizing marginal likelihood; understand empirical Bayes approaches.

Hyperparameters and Evidence Approximation

What You'll Learn

You'll learn how to tune hyperparameters like prior precision and noise variance by maximizing the marginal likelihood, a key technique in empirical Bayes.

The Core Idea

In Bayesian learning, we've seen how to place priors on model weights. But those priors themselves have hyperparameters—for example, the precision (inverse variance) of a Gaussian prior, or the noise precision in our likelihood function.

Instead of guessing these values, we can learn them from data by maximizing the marginal likelihood (also called evidence). The marginal likelihood is the probability of observing your data, averaging over all possible weight values:

p(D | α, β) = ∫ p(D | w, β) p(w | α) dw

Where α might be prior precision and β noise precision.

Empirical Bayes is the approach where we treat these hyperparameters as tunable parameters rather than choosing them subjectively or using full hierarchical Bayes (which would put priors on the hyperparameters too). We find the α and β that make our observed data most probable.

Why This Matters

Think of it like adjusting your camera settings: you want enough regularization (controlled by prior precision α) to prevent overfitting, but not so much that you ignore the data. The noise precision β tells you how much to trust each data point. By maximizing evidence, the model automatically finds this balance—loose priors when data is plentiful and informative, tight priors when data is scarce.

This is closely related to Maximum A Posteriori Estimation you just learned, but now we're optimizing at the hyperparameter level, not the weight level.

Key Takeaway: Evidence approximation tunes hyperparameters (like prior and noise precision) by maximizing marginal likelihood, letting the data guide how much regularization to apply—a pragmatic middle ground between pure Bayesian and frequentist approaches.