KL Annealing Strategies
What you'll learn: How gradually increasing the weight of the KL divergence term during VAE training prevents posterior collapse and improves model quality.
The Problem: Balancing Two Objectives
Remember that VAEs optimize two competing goals: reconstruction accuracy (making outputs match inputs) and latent space regularization (keeping the posterior close to the prior via KL divergence). When training starts, the model often ignores the latent space entirely, collapsing to a trivial solution where the encoder outputs the prior distribution directly—this is posterior collapse.
What Is KL Annealing?
KL annealing means starting with a very small (or zero) weight on the KL divergence term and gradually increasing it to its full value (typically 1.0) over the course of training. Instead of optimizing:
Loss = Reconstruction_Loss + KL_Divergence
You optimize:
Loss = Reconstruction_Loss + β(t) * KL_Divergence
where β(t) starts near 0 and increases to 1 over time (often linearly or following a sigmoid schedule).
Why It Works
Think of it like learning to ride a bike with training wheels. Early in training, the model focuses almost entirely on reconstruction—learning useful representations without the pressure of regularization. As training progresses and β(t) increases, the model gradually learns to organize its latent space properly. By the time β = 1, the encoder has already discovered meaningful features and won't abandon them.
Key Takeaway: KL annealing schedules the weight of the KL term from near-zero to full strength during training, allowing the VAE to first learn useful latent representations before enforcing prior matching, effectively preventing posterior collapse.