Course contentsShow
← Machine Learning and Deep Learning
Lesson 1509 of 3,53833. Generative Adversarial NetworksFree lesson

Two-Timescale Update Rule

Using different update frequencies or learning rates for discriminator and generator to balance training dynamics.

Two-Timescale Update Rule

What you'll learn: How using different learning rates or update frequencies for the discriminator and generator can stabilize GAN training.

The Core Problem

In GAN training, the discriminator and generator play a competitive game. However, they don't always learn at the same pace. If one player becomes too strong too quickly, the game becomes unbalanced—like playing chess against someone far more skilled. The weaker player can't improve because they're overwhelmed.

What Is Two-Timescale Update Rule (TTUR)?

The Two-Timescale Update Rule addresses this by deliberately updating the discriminator and generator at different speeds. Think of it as giving one player more practice time to keep the match competitive.

Common approaches:

  1. Different learning rates: Set the discriminator's learning rate lower than the generator's (e.g., lr_D = 0.0001, lr_G = 0.0004)
  2. Different update frequencies: Update the discriminator multiple times per generator update (e.g., 5 discriminator steps for every 1 generator step)

Why This Helps

The discriminator typically learns faster because its task (classifying real vs. fake) is simpler than the generator's task (creating realistic samples). Without TTUR:

  • The discriminator becomes too strong, providing gradients that are either too weak or unhelpful
  • The generator can't improve, leading to training instability or mode collapse

By slowing down the discriminator or speeding up the generator, you maintain a healthy competitive balance where both networks can learn effectively.

Practical Impact

TTUR helps achieve convergence even when the networks aren't perfectly matched in capacity. It's especially useful when combined with other stabilization techniques like spectral normalization or gradient penalties you've already learned.

Key Takeaway: Two-Timescale Update Rule balances GAN training by updating the discriminator and generator at different speeds (via learning rates or frequencies), preventing one network from overpowering the other and enabling stable convergence.