VAE as Variational Inference
What You'll Learn
You'll understand how VAEs use variational inference to approximate an intractable posterior distribution with a learnable, tractable one.
The Core Problem
In generative modeling, we want to know p(z|x) — the probability of latent variables z given observed data x. This tells us "what hidden representation likely generated this data?"
Unfortunately, computing p(z|x) directly requires integrating over all possible latent values:
p(z|x) = p(x|z)p(z) / p(x), where p(x) = ∫ p(x|z)p(z) dz
That integral is intractable for complex models — we can't compute it.
The Variational Solution
Variational inference solves this by approximation:
Instead of computing the true posterior p(z|x), we introduce a simpler, parametric distribution q(z|x) that we can compute and optimize. In VAEs, this is the encoder network.
Think of it like approximating a complex mountain range (true posterior) with a smooth, simple hill (q) that's close enough for practical purposes.
How VAEs Fit In
The VAE encoder outputs parameters of q(z|x) — typically mean μ and variance σ² of a Gaussian. The decoder represents p(x|z).
By minimizing the ELBO (which you've seen), we're simultaneously:
- Making q(z|x) close to the true p(z|x) (via KL divergence)
- Ensuring the latent codes decode well (via reconstruction loss)
This is variational inference in action: we're learning the best tractable approximation to an intractable distribution.
Key Takeaway: VAEs frame generative modeling as variational inference, replacing the intractable true posterior p(z|x) with a learnable approximate posterior q(z|x) parameterized by the encoder network.