VAE Training Loop in PyTorch
What You'll Learn
You'll learn how to bring together all VAE components into a complete training loop that optimizes the ELBO objective.
The Complete Training Cycle
Now that you've built the encoder, decoder, implemented the reparameterization trick, and understood ELBO, it's time to connect everything into a working training loop.
The VAE training loop follows the standard PyTorch pattern but with a custom loss calculation. Each iteration involves:
- Forward Pass: Feed input through encoder to get mean and log-variance, sample latent code using reparameterization, decode to reconstruct input
- ELBO Loss Calculation: Compute reconstruction loss (how well you rebuild the input) plus KL divergence (how much your posterior deviates from the prior)
- Backward Pass: Compute gradients with respect to all parameters
- Optimizer Step: Update encoder and decoder weights
Think of it like teaching someone to draw: they look at a picture (encoder), capture its essence in memory (latent space), then redraw it (decoder). The training loop repeatedly adjusts their "drawing skills" to both reproduce accurately and keep their mental representation organized.
optimizer.zero_grad()
# Forward pass
mu, logvar = encoder(x)
z = reparameterize(mu, logvar)
x_recon = decoder(z)
# ELBO loss
recon_loss = reconstruction_loss(x_recon, x)
kl_loss = kl_divergence(mu, logvar)
loss = recon_loss + beta * kl_loss
# Backward pass
loss.backward()
# Update weights
optimizer.step()
The key is that all components are differentiable, allowing gradients to flow from the loss back through the entire architecture.
Key Takeaway: The VAE training loop orchestrates the forward pass (encode → sample → decode), ELBO computation (reconstruction + KL), backpropagation, and weight updates in each iteration, training both encoder and decoder end-to-end.