Measuring Training Stability
What you'll learn: How to monitor key signals during GAN training to catch instability before it derails your model.
Why Stability Monitoring Matters
Training GANs is like watching two opponents in a delicate dance—when one stumbles, both fall. Unlike standard neural networks where loss steadily decreases, GAN training involves competing objectives that can spiral out of control. You need early warning systems to catch problems before they become catastrophic.
Core Signals to Track
Loss Trajectories
Monitor both discriminator loss and generator loss over time. Healthy training shows:
- Oscillating but bounded losses: They should fluctuate but stay within a reasonable range
- No divergence: Losses shouldn't shoot toward infinity or collapse to zero
- Rough balance: Neither network should completely dominate (though exact equality isn't required)
Warning Signs
Watch for these red flags:
- Generator loss increasing monotonically: The discriminator is winning too easily
- Discriminator loss approaching zero: It's becoming too confident, starving the generator of gradients
- Wild oscillations: Losses swinging dramatically suggest unstable dynamics
- Flat generator loss with high discriminator loss: Often indicates mode collapse
Additional Stability Metrics
Beyond raw losses, track:
- Gradient norms: Sudden spikes or vanishing values signal trouble
- Discriminator confidence: Average output on real vs. fake samples
- Generated sample diversity: Visual inspection or automated metrics
Practical Monitoring
Log these metrics every few iterations. Plot them during training. If you see divergence patterns emerging, you can intervene with learning rate adjustments, architecture tweaks, or regularization before wasting hours on a doomed run.
Key Takeaway: Stable GAN training requires actively monitoring discriminator and generator losses alongside gradient norms and output diversity—catching instability early lets you intervene before training completely fails.