Course contentsShow
← Machine Learning and Deep Learning
Lesson 1508 of 3,53833. Generative Adversarial NetworksFree lesson

Spectral Normalization

Normalizing weight matrices by their spectral norm to control discriminator Lipschitz constant and stabilize training.

Spectral Normalization

What you'll learn: How to stabilize GAN training by constraining the discriminator's weight matrices to prevent gradients from exploding.

The Core Problem

In GANs, the discriminator can become too powerful too quickly, causing training instability. One root cause is that the discriminator's weights can grow large, leading to unpredictable gradient magnitudes that destabilize the generator's learning.

What Is Spectral Normalization?

Spectral normalization is a technique that normalizes each weight matrix in your discriminator by dividing it by its spectral norm—the largest singular value of that matrix. This effectively caps how much any layer can amplify its input, keeping the discriminator's behavior smooth and predictable.

Think of it like installing speed limiters on every layer of your discriminator. Instead of allowing individual layers to amplify signals by arbitrary amounts (which can compound through the network), you ensure each layer has a controlled, bounded influence.

Why It Works

By normalizing weights this way, you control the Lipschitz constant of the discriminator—essentially limiting how rapidly the discriminator's output can change in response to input changes. This constraint:

  • Prevents discriminator gradients from vanishing or exploding
  • Allows the generator to receive more stable, useful feedback
  • Reduces mode collapse and training oscillations

Unlike gradient penalties (which you've seen), spectral normalization works directly on the weights before forward passes, adding minimal computational overhead.

Practical Impact

Spectral normalization has become a standard ingredient in modern GAN architectures because it "just works"—apply it to discriminator layers, and training becomes noticeably more stable without careful hyperparameter tuning.

Key Takeaway: Spectral normalization stabilizes GAN training by normalizing discriminator weight matrices by their spectral norm, controlling the Lipschitz constant and preventing gradient pathologies.