The ELBO Objective in Practice
What you'll learn: How to actually compute and optimize the Evidence Lower Bound (ELBO) during VAE training, balancing reconstruction quality and latent regularization.
Computing ELBO in Your Training Loop
You've learned that the ELBO combines reconstruction loss and KL divergence, but how do you actually implement this during training?
In practice, you compute both terms for each batch and sum them (often weighted). Here's the workflow:
- Encode: Pass data through the probabilistic encoder to get mean (
μ) and log-variance (log_σ²) - Sample: Use the reparameterization trick:
z = μ + σ * εwhereε ~ N(0,1) - Decode: Generate reconstruction from
z - Compute reconstruction loss: Measure how well the decoder reconstructed the input (e.g., MSE or binary cross-entropy)
- Compute KL divergence: Calculate
KL(q(z|x) || p(z))analytically (closed form exists for Gaussian prior) - Combine:
loss = reconstruction_loss + β * kl_divergence
The Balancing Act
Think of ELBO optimization like tuning a piano: the reconstruction term pulls toward perfect copies, while the KL term keeps the latent space organized and smooth. Too much KL weight and you get blurry reconstructions but nice latent structure. Too little and you get sharp reconstructions but chaotic, unusable latent spaces.
The coefficient β (from β-VAE) controls this trade-off. Start with β = 1.0 (standard VAE), then adjust based on whether you prioritize generation quality or disentanglement.
Key Takeaway: In practice, optimizing the ELBO means computing reconstruction loss and KL divergence separately for each batch, then combining them with appropriate weighting—monitor both terms during training to diagnose issues like posterior collapse or poor reconstruction.