Course contentsShow
← Machine Learning and Deep Learning
Lesson 581 of 3,53814. Probabilistic and Bayesian Machine LearningFree lesson

Limitations of Exact Inference

Explore why exact inference becomes intractable in high dimensions and complex models, motivating approximate methods.

Limitations of Exact Inference

What You'll Learn

You'll discover why exact Bayesian inference—computing posterior distributions analytically—quickly becomes impossible for real-world models, setting the stage for approximate methods.

The Promise and the Problem

In previous lessons, you learned that exact inference lets us compute exact posterior distributions using conjugate priors and analytical formulas. This is elegant: given data, you can marginalize over all parameter values and get precise uncertainty estimates.

But there's a catch: exact inference only works in very specific, simple scenarios.

Why Exact Inference Breaks Down

High-Dimensional Parameter Spaces

Imagine a neural network with thousands or millions of weights. Computing the posterior requires integrating over all possible combinations of these parameters. That's an integral in millions of dimensions—analytically impossible and computationally infeasible.

Non-Conjugate Priors

When your prior and likelihood aren't conjugate pairs, you can't derive a closed-form posterior. You're stuck with an integral that has no analytical solution.

Complex Likelihood Functions

Deep learning models, hierarchical structures, and non-linear relationships create likelihood functions that resist analytical treatment. The math simply doesn't simplify.

Discrete and Mixed Spaces

Many real models involve discrete choices (which layer architecture?) combined with continuous parameters (what weight values?). Exact marginalization across these mixed spaces is intractable.

The Analogy

Think of exact inference like hand-calculating every possible route on a map to find the shortest path. For a small neighborhood, it's doable. For an entire country with millions of roads? You need approximations—shortcuts, heuristics, or sampling methods.

Key Takeaway: Exact inference is limited to low-dimensional models with conjugate structure; real-world complexity demands approximate inference methods like variational inference and Markov Chain Monte Carlo (MCMC).