The VAE Loss Function: ELBO
You'll learn how VAEs combine two complementary objectives—reconstruction accuracy and regularization—into a single training loss called the Evidence Lower Bound.
What Is ELBO?
The Evidence Lower Bound (ELBO) is the loss function that makes VAEs work. While basic autoencoders only care about reconstructing inputs accurately, VAEs add a second goal: ensuring the latent space follows a well-behaved probability distribution (typically standard normal). ELBO elegantly balances both objectives.
The Two Components
ELBO consists of two terms:
-
Reconstruction Loss: Measures how well the decoder reconstructs the original input from the sampled latent code. This is identical to what you've seen in basic autoencoders—typically MSE for continuous data or binary cross-entropy for images.
-
KL Divergence: Measures how much the encoder's output distribution
q(z|x)diverges from the prior distributionp(z)(usually standard normal). This acts as a regularizer, preventing the encoder from learning arbitrary, chaotic latent representations.
Why "Lower Bound"?
In probability theory, we want to maximize the likelihood of our data. Direct computation is intractable, so ELBO provides a lower bound we can actually optimize. Maximizing ELBO is equivalent to minimizing the VAE loss:
VAE Loss = Reconstruction Loss + β × KL Divergence
The hyperparameter β controls the trade-off. Higher β pushes latent codes toward the prior but may sacrifice reconstruction quality.
The Balancing Act
Think of ELBO as training a compression system with quality control: reconstruction loss ensures fidelity, while KL divergence ensures the compressed representation stays organized and interpretable—enabling smooth interpolation and meaningful sampling.
Key Takeaway: ELBO combines reconstruction loss (decoder quality) and KL divergence (encoder regularization) into a single objective that trains VAEs to learn structured, generative latent spaces.