Learning Rate Balance
What You'll Learn
You'll learn how balancing the learning rates of the generator and discriminator prevents one network from overpowering the other during GAN training.
The Core Problem
In GAN training, two networks play a competitive game: the generator tries to fool the discriminator, while the discriminator tries to detect fakes. If one network learns much faster than the other, the game becomes unfair and training breaks down.
Think of it like a sparring match between two fighters. If one improves much faster than the other, the weaker fighter stops learning anything useful—they're either always winning too easily or losing too badly to adapt.
Why Learning Rate Balance Matters
When the discriminator learns too fast, it becomes perfect at detecting fakes. The generator receives only vanishing gradients (no useful feedback), so it stops improving—leading to mode collapse or stagnation.
When the generator learns too fast, it exploits weaknesses in a weak discriminator. The discriminator can't keep up, providing meaningless feedback. The generator produces low-quality samples that temporarily fool the naive discriminator.
Finding the Balance
The key is tuning learning rates so both networks improve together:
- Common starting point: Use the same learning rate for both (e.g.,
0.0002) - Monitor training: Watch for signs one network is dominating (discriminator loss near 0 or 1, generator loss exploding)
- Adjust carefully: Lower the learning rate of the stronger network or raise the weaker one
- Alternative: Train discriminator fewer times per generator update (e.g., 1:1 ratio instead of 5:1)
The goal is maintaining tension in the minimax game—each network should win roughly half the time during healthy training.
Key Takeaway: Balanced learning rates keep the generator and discriminator evenly matched, ensuring both networks receive meaningful gradients and continue improving throughout training.