Course contentsShow
← Machine Learning and Deep Learning
Lesson 1479 of 3,53833. Generative Adversarial NetworksFree lesson

Vanishing Gradients in GANs

How an overpowered discriminator can cause generator gradients to vanish, stopping learning.

Vanishing Gradients in GANs

What you'll learn: How an overpowered discriminator can cause generator gradients to vanish, stopping learning.

The Problem

In GANs, the discriminator and generator compete in a minimax game. But when the discriminator becomes too good too fast—perfectly distinguishing real from fake—it creates a serious training problem: vanishing gradients for the generator.

Why This Happens

Think of the discriminator as a teacher grading the generator's work. If the teacher becomes an expert instantly and starts giving every assignment a hard F (probability near 0 for fake images), the generator receives almost no useful feedback. The gradient signal—which tells the generator how to improve—becomes extremely small, approaching zero.

Mathematically, when the discriminator outputs values very close to 0 or 1 with high confidence, the log functions in the GAN objective flatten out. Their derivatives become tiny, meaning the generator gets almost no gradient to update its weights.

The Consequence

When gradients vanish:

  • The generator stops learning or learns extremely slowly
  • Training appears to "stall" despite continuing iterations
  • The discriminator wins the game too easily
  • Mode collapse often follows, as the generator can't explore new strategies

This is why training stability challenges are so critical in GANs—you need careful balancing between discriminator and generator updates to prevent either from dominating too quickly.

Key Takeaway: When the discriminator becomes too powerful, it outputs near-perfect classifications that produce vanishingly small gradients, starving the generator of the learning signal it needs to improve.