BigGAN: Scaling Up GAN Training
What you'll learn: How BigGAN achieves state-of-the-art image generation on ImageNet by combining large batch sizes, orthogonal regularization, and truncation tricks.
The Core Innovation
BigGAN demonstrated that GANs could scale dramatically when you throw more compute at them—but only if you handle the instabilities that come with scale. Think of it like turning up the volume on an amplifier: you get louder music, but also more distortion unless you carefully tune other settings.
Three Key Techniques
Large Batch Sizes
BigGAN uses batch sizes 2-8× larger than typical GANs (often 256-2048 samples). Larger batches provide more stable gradient estimates, like polling more people for a more reliable survey result. However, this requires careful learning rate tuning to maintain training dynamics.
Orthogonal Regularization
To prevent weights from becoming too correlated (which causes instability), BigGAN applies orthogonal regularization to the generator's weight matrices. This encourages weights to remain "independent" from each other, like keeping different instruments in an orchestra from drowning each other out.
Truncation Trick
At generation time, BigGAN samples latent vectors from a truncated normal distribution (e.g., cutting off values beyond 2 standard deviations). This trades diversity for quality—you get fewer "weird" samples but much higher fidelity images. It's like choosing to paint only landscapes you've seen before rather than imagining alien worlds.
The Result
These techniques enabled BigGAN to generate 512×512 ImageNet images with unprecedented realism, achieving breakthrough Inception Score and FID metrics. The approach proved that with proper stabilization, GANs could scale to complex, diverse datasets.
Key Takeaway: BigGAN shows that scaling GAN training requires three pillars: large batches for stable gradients, orthogonal regularization for weight diversity, and truncation for quality-focused sampling.