Conjugate Priors and Analytical Posteriors
What you'll learn: How choosing the right prior family lets you compute exact posterior distributions without numerical approximation.
The Core Idea
In Bayesian inference, you start with a prior distribution over parameters, observe data (the likelihood), and use Bayes' rule to compute the posterior distribution. Normally, this involves tricky integrals that require approximation methods. But for certain special prior-likelihood pairs, the posterior has the same distributional form as the prior—these are called conjugate priors.
Think of it like LEGO blocks that snap together perfectly: if your prior and likelihood are "compatible shapes," the resulting posterior is also the same shape, just with updated parameters.
Why This Matters
When a prior is conjugate to the likelihood:
- The posterior can be computed in closed form (exact formula, no sampling needed)
- You get analytical expressions for posterior parameters
- Updates are fast and exact—perfect for online learning or real-time inference
Classic Examples
- Beta-Binomial conjugacy: If you have a Beta prior on probability and observe Binomial data (coin flips), the posterior is also Beta
- Gaussian-Gaussian conjugacy: With a Gaussian prior on the mean and Gaussian likelihood, the posterior mean is also Gaussian
- Gamma-Poisson conjugacy: Gamma prior + Poisson likelihood → Gamma posterior
The Trade-off
Conjugate priors are mathematically convenient but might not perfectly capture your true prior beliefs. You're choosing analytical tractability over complete modeling flexibility.
Key Takeaway: Conjugate priors let you perform exact Bayesian inference with closed-form posteriors, avoiding expensive numerical methods—but only for specific prior-likelihood combinations that "close under" Bayes' rule.