Course contentsShow
← Machine Learning and Deep Learning
Lesson 1457 of 3,53832. Generative Models: Autoencoders and VAEsFree lesson

The ELBO Objective in Practice

How to compute and optimize the Evidence Lower Bound during VAE training, balancing reconstruction and KL terms.

The ELBO Objective in Practice

What you'll learn: How to actually compute and optimize the Evidence Lower Bound (ELBO) during VAE training, balancing reconstruction quality and latent regularization.

Computing ELBO in Your Training Loop

You've learned that the ELBO combines reconstruction loss and KL divergence, but how do you actually implement this during training?

In practice, you compute both terms for each batch and sum them (often weighted). Here's the workflow:

  1. Encode: Pass data through the probabilistic encoder to get mean (μ) and log-variance (log_σ²)
  2. Sample: Use the reparameterization trick: z = μ + σ * ε where ε ~ N(0,1)
  3. Decode: Generate reconstruction from z
  4. Compute reconstruction loss: Measure how well the decoder reconstructed the input (e.g., MSE or binary cross-entropy)
  5. Compute KL divergence: Calculate KL(q(z|x) || p(z)) analytically (closed form exists for Gaussian prior)
  6. Combine: loss = reconstruction_loss + β * kl_divergence

The Balancing Act

Think of ELBO optimization like tuning a piano: the reconstruction term pulls toward perfect copies, while the KL term keeps the latent space organized and smooth. Too much KL weight and you get blurry reconstructions but nice latent structure. Too little and you get sharp reconstructions but chaotic, unusable latent spaces.

The coefficient β (from β-VAE) controls this trade-off. Start with β = 1.0 (standard VAE), then adjust based on whether you prioritize generation quality or disentanglement.

Key Takeaway: In practice, optimizing the ELBO means computing reconstruction loss and KL divergence separately for each batch, then combining them with appropriate weighting—monitor both terms during training to diagnose issues like posterior collapse or poor reconstruction.