Course contentsShow
← Machine Learning and Deep Learning
Lesson 1464 of 3,53832. Generative Models: Autoencoders and VAEsFree lesson

KL Annealing Strategies

Gradually increasing the KL weight during training to prevent posterior collapse and improve reconstruction quality.

KL Annealing Strategies

What you'll learn: How gradually increasing the weight of the KL divergence term during VAE training prevents posterior collapse and improves model quality.

The Problem: Balancing Two Objectives

Remember that VAEs optimize two competing goals: reconstruction accuracy (making outputs match inputs) and latent space regularization (keeping the posterior close to the prior via KL divergence). When training starts, the model often ignores the latent space entirely, collapsing to a trivial solution where the encoder outputs the prior distribution directly—this is posterior collapse.

What Is KL Annealing?

KL annealing means starting with a very small (or zero) weight on the KL divergence term and gradually increasing it to its full value (typically 1.0) over the course of training. Instead of optimizing:

Loss = Reconstruction_Loss + KL_Divergence

You optimize:

Loss = Reconstruction_Loss + β(t) * KL_Divergence

where β(t) starts near 0 and increases to 1 over time (often linearly or following a sigmoid schedule).

Why It Works

Think of it like learning to ride a bike with training wheels. Early in training, the model focuses almost entirely on reconstruction—learning useful representations without the pressure of regularization. As training progresses and β(t) increases, the model gradually learns to organize its latent space properly. By the time β = 1, the encoder has already discovered meaningful features and won't abandon them.

Key Takeaway: KL annealing schedules the weight of the KL term from near-zero to full strength during training, allowing the VAE to first learn useful latent representations before enforcing prior matching, effectively preventing posterior collapse.