Course contentsShow
← Machine Learning and Deep Learning
Lesson 1448 of 3,53832. Generative Models: Autoencoders and VAEsFree lesson

Deriving the VAE Objective

Follow the mathematical derivation from log-likelihood to the ELBO via Jensen's inequality.

Deriving the VAE Objective

What you'll learn: How to mathematically arrive at the ELBO (Evidence Lower Bound) objective that VAEs actually optimize, starting from the log-likelihood we'd ideally maximize.

From Likelihood to ELBO

When training a VAE, we want to maximize the log-likelihood of our data: log p(x). The problem? Computing p(x) requires integrating over all possible latent codes z, which is intractable:

p(x) = ∫ p(x|z) p(z) dz

This integral is impossibly expensive in high dimensions.

Introducing the Approximate Posterior

Instead, we introduce an approximate posterior q(z|x) — our encoder — to help. We can write:

log p(x) = log ∫ p(x|z) p(z) dz
         = log ∫ [p(x|z) p(z) / q(z|x)] q(z|x) dz
         = log E_q(z|x) [p(x|z) p(z) / q(z|x)]

Applying Jensen's Inequality

Here's the key trick: Jensen's inequality says that for a concave function like log, the log of an expectation is ≥ the expectation of the log:

log E[X] ≥ E[log X]

Applying this:

log p(x) ≥ E_q(z|x) [log p(x|z) + log p(z) - log q(z|x)]
        = E_q(z|x) [log p(x|z)] - KL(q(z|x) || p(z))

This right-hand side is the ELBO (Evidence Lower Bound) — a lower bound on the log-likelihood that's tractable to compute and optimize!

Analogy: Think of it like estimating a mountain's height from below. We can't measure the peak directly (log p(x)), but we can build a lower estimate (ELBO) that gets us close enough.

Key Takeaway: The VAE objective (ELBO) emerges by applying Jensen's inequality to the intractable log-likelihood, giving us a tractable lower bound containing reconstruction loss and KL divergence terms.