Course contentsShow
← Machine Learning and Deep Learning
Lesson 1459 of 3,53832. Generative Models: Autoencoders and VAEsFree lesson

KL Divergence Computation for Gaussian Latents

Deriving and implementing the closed-form KL divergence between learned and prior distributions in code.

KL Divergence Computation for Gaussian Latents

What You'll Learn

You'll learn how to derive and implement the closed-form formula for computing KL divergence between your VAE's learned Gaussian distribution and the standard normal prior.

The Core Idea

In VAEs, we need to measure how much the encoder's predicted distribution (with learned mean μ and variance σ²) differs from our chosen prior (standard normal: mean=0, variance=1). This is the KL divergence regularization term.

The beautiful news: for Gaussian distributions, we don't need numerical approximation. There's a closed-form solution we can compute directly!

The Formula

For each latent dimension, the KL divergence between N(μ, σ²) and N(0, 1) is:

KL = 0.5 * (σ² + μ² - 1 - log(σ²))

Why this matters: This formula is fast, differentiable, and exact—no sampling needed during the KL computation itself.

Implementation in PyTorch

def kl_divergence_gaussian(mu, logvar):
    """
    Compute KL(q(z|x) || p(z)) for Gaussian latents.
    
    Args:
        mu: predicted mean from encoder [batch_size, latent_dim]
        logvar: predicted log variance [batch_size, latent_dim]
    
    Returns:
        KL divergence [batch_size] (summed over latent dimensions)
    """
    # KL = 0.5 * sum(σ² + μ² - 1 - log(σ²))
    # Since we store log(σ²), exp(logvar) = σ²
    kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp(), dim=1)
    return kl

Key Takeaway: The closed-form KL divergence for Gaussian latents is 0.5 * (σ² + μ² - 1 - log(σ²)), which we compute efficiently using the encoder's predicted mean and log-variance without additional sampling.