Course contentsShow
← Machine Learning and Deep Learning
Lesson 1501 of 3,53833. Generative Adversarial NetworksFree lesson

Non-Convergent Dynamics

Understanding why GAN training is a minimax game rather than simple optimization and its convergence challenges.

Non-Convergent Dynamics

What you'll learn: Why GAN training often fails to settle into a stable solution, unlike typical neural network optimization.

The Minimax Game Difference

In standard deep learning, you minimize a single loss function—imagine rolling a ball down a hill until it reaches the bottom. The ball always converges to a stable point.

GANs are fundamentally different. They're a minimax game between two players (generator and discriminator), each with opposing goals. The generator tries to maximize the discriminator's error, while the discriminator tries to minimize it. This creates dynamics more like two opponents circling each other in a boxing ring rather than a ball rolling downhill.

Why Convergence Fails

In game theory, stable solutions are called Nash equilibria. GANs theoretically have one: when the generator produces perfect real-looking samples and the discriminator can only guess randomly. However, reaching this equilibrium is extremely difficult because:

  1. Oscillating Updates: When you update the generator, you change what the discriminator should learn. When you update the discriminator, you change the generator's training signal. They chase each other in circles rather than converging.

  2. No Single Loss Surface: Unlike standard optimization where you descend a fixed landscape, GANs have a constantly shifting terrain. Each player's update changes the problem for the other.

  3. Sensitivity to Hyperparameters: The learning rates, update frequencies, and architecture choices critically affect whether the game stabilizes or spirals out of control.

This is why GAN training requires careful balancing tricks—you're not just optimizing, you're mediating a competitive game that may never truly settle.

Key Takeaway: GANs use minimax game dynamics rather than simple optimization, causing training to oscillate or diverge instead of converging smoothly to a solution like traditional neural networks.