Vanishing Gradients in GANs
What you'll learn: How discriminator dominance starves the generator of learning signals, creating a unique gradient problem in adversarial training.
The Core Problem
In GANs, vanishing gradients occur when the discriminator becomes too good too fast. When the discriminator perfectly distinguishes real from fake images, it outputs probabilities very close to 0 for generated images and 1 for real ones. These extreme predictions push outputs through saturated regions of the sigmoid function, where gradients become vanishingly small.
Here's the critical difference from standard networks: the generator only learns through the discriminator's gradients. If those gradients vanish, the generator receives no useful learning signal—it's like trying to navigate in complete darkness with no feedback.
Why This Differs from Standard Networks
In traditional deep networks, vanishing gradients happen when many layers multiply small gradients during backpropagation, weakening the signal as it travels backward.
In GANs, vanishing gradients arise from the adversarial dynamic itself:
- A dominant discriminator creates extremely confident classifications
- The generator's loss becomes flat (saturated) because fake samples are consistently rejected
- Small gradient → generator can't update meaningfully → can't improve → discriminator stays dominant
Think of it like a talent show where the judge becomes impossibly harsh. If every performance gets a flat "0/10," contestants receive no actionable feedback to improve—they're stuck.
Why It Matters
This isn't just slow training—it's a training failure mode. The generator stops learning entirely while the discriminator continues improving, breaking the minimax game balance that GANs rely on.
Key Takeaway: Vanishing gradients in GANs occur when a too-powerful discriminator produces saturated outputs, eliminating the generator's learning signal—a problem rooted in adversarial dynamics rather than network depth.