KL Divergence Computation for Gaussian Latents
What You'll Learn
You'll learn how to derive and implement the closed-form formula for computing KL divergence between your VAE's learned Gaussian distribution and the standard normal prior.
The Core Idea
In VAEs, we need to measure how much the encoder's predicted distribution (with learned mean μ and variance σ²) differs from our chosen prior (standard normal: mean=0, variance=1). This is the KL divergence regularization term.
The beautiful news: for Gaussian distributions, we don't need numerical approximation. There's a closed-form solution we can compute directly!
The Formula
For each latent dimension, the KL divergence between N(μ, σ²) and N(0, 1) is:
KL = 0.5 * (σ² + μ² - 1 - log(σ²))
Why this matters: This formula is fast, differentiable, and exact—no sampling needed during the KL computation itself.
Implementation in PyTorch
def kl_divergence_gaussian(mu, logvar):
"""
Compute KL(q(z|x) || p(z)) for Gaussian latents.
Args:
mu: predicted mean from encoder [batch_size, latent_dim]
logvar: predicted log variance [batch_size, latent_dim]
Returns:
KL divergence [batch_size] (summed over latent dimensions)
"""
# KL = 0.5 * sum(σ² + μ² - 1 - log(σ²))
# Since we store log(σ²), exp(logvar) = σ²
kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp(), dim=1)
return kl
Key Takeaway: The closed-form KL divergence for Gaussian latents is 0.5 * (σ² + μ² - 1 - log(σ²)), which we compute efficiently using the encoder's predicted mean and log-variance without additional sampling.