Deriving the VAE Objective
What you'll learn: How to mathematically arrive at the ELBO (Evidence Lower Bound) objective that VAEs actually optimize, starting from the log-likelihood we'd ideally maximize.
From Likelihood to ELBO
When training a VAE, we want to maximize the log-likelihood of our data: log p(x). The problem? Computing p(x) requires integrating over all possible latent codes z, which is intractable:
p(x) = ∫ p(x|z) p(z) dz
This integral is impossibly expensive in high dimensions.
Introducing the Approximate Posterior
Instead, we introduce an approximate posterior q(z|x) — our encoder — to help. We can write:
log p(x) = log ∫ p(x|z) p(z) dz
= log ∫ [p(x|z) p(z) / q(z|x)] q(z|x) dz
= log E_q(z|x) [p(x|z) p(z) / q(z|x)]
Applying Jensen's Inequality
Here's the key trick: Jensen's inequality says that for a concave function like log, the log of an expectation is ≥ the expectation of the log:
log E[X] ≥ E[log X]
Applying this:
log p(x) ≥ E_q(z|x) [log p(x|z) + log p(z) - log q(z|x)]
= E_q(z|x) [log p(x|z)] - KL(q(z|x) || p(z))
This right-hand side is the ELBO (Evidence Lower Bound) — a lower bound on the log-likelihood that's tractable to compute and optimize!
Analogy: Think of it like estimating a mountain's height from below. We can't measure the peak directly (log p(x)), but we can build a lower estimate (ELBO) that gets us close enough.
Key Takeaway: The VAE objective (ELBO) emerges by applying Jensen's inequality to the intractable log-likelihood, giving us a tractable lower bound containing reconstruction loss and KL divergence terms.