Course contentsShow
← Machine Learning and Deep Learning
Lesson 1465 of 3,53832. Generative Models: Autoencoders and VAEsFree lesson

Posterior Collapse and Solutions

Recognizing when the model ignores latent codes, and techniques like free bits to enforce meaningful latent usage.

Posterior Collapse and Solutions

What you'll learn: You'll understand why VAEs sometimes ignore their latent codes and how to force them to use those codes meaningfully.

The Problem: When Your VAE Gets Lazy

Posterior collapse happens when your VAE's decoder becomes so powerful that it learns to generate good reconstructions without using information from the latent code z. The encoder still produces latent variables, but the decoder ignores them—the KL divergence term drops to nearly zero, and the latent space becomes uninformative.

Think of it like a student doing a group project: if one member (the decoder) is extremely capable, they might just do everything themselves and ignore their partner's (the encoder's) input entirely. The project succeeds, but collaboration fails.

Why It Happens

Strong autoregressive decoders (like LSTMs or Transformers) can model data distributions well on their own. When the KL penalty pushes the posterior toward the prior (a simple Gaussian), the decoder finds it easier to ignore the noisy latent code and rely solely on its own modeling power.

Solutions

Free Bits: Reserve a minimum amount of "information capacity" for each latent dimension. You only penalize KL divergence if it drops below a threshold (e.g., 2 bits per dimension), ensuring the model must use the latent space.

KL Annealing: Gradually increase the KL weight during training (you've seen this already). This gives the encoder time to learn useful representations before the regularization pressure kicks in fully.

Weakening the Decoder: Use simpler decoder architectures or add noise to decoder inputs, forcing reliance on latent information.

Key Takeaway: Posterior collapse occurs when the decoder ignores latent codes; techniques like free bits enforce a minimum information threshold, ensuring your VAE learns meaningful latent representations rather than bypassing them entirely.