Limitations of Exact Inference
What You'll Learn
You'll discover why exact Bayesian inference—computing posterior distributions analytically—quickly becomes impossible for real-world models, setting the stage for approximate methods.
The Promise and the Problem
In previous lessons, you learned that exact inference lets us compute exact posterior distributions using conjugate priors and analytical formulas. This is elegant: given data, you can marginalize over all parameter values and get precise uncertainty estimates.
But there's a catch: exact inference only works in very specific, simple scenarios.
Why Exact Inference Breaks Down
High-Dimensional Parameter Spaces
Imagine a neural network with thousands or millions of weights. Computing the posterior requires integrating over all possible combinations of these parameters. That's an integral in millions of dimensions—analytically impossible and computationally infeasible.
Non-Conjugate Priors
When your prior and likelihood aren't conjugate pairs, you can't derive a closed-form posterior. You're stuck with an integral that has no analytical solution.
Complex Likelihood Functions
Deep learning models, hierarchical structures, and non-linear relationships create likelihood functions that resist analytical treatment. The math simply doesn't simplify.
Discrete and Mixed Spaces
Many real models involve discrete choices (which layer architecture?) combined with continuous parameters (what weight values?). Exact marginalization across these mixed spaces is intractable.
The Analogy
Think of exact inference like hand-calculating every possible route on a map to find the shortest path. For a small neighborhood, it's doable. For an entire country with millions of roads? You need approximations—shortcuts, heuristics, or sampling methods.
Key Takeaway: Exact inference is limited to low-dimensional models with conjugate structure; real-world complexity demands approximate inference methods like variational inference and Markov Chain Monte Carlo (MCMC).