The Lottery Ticket Hypothesis
What You'll Learn
You'll discover a surprising finding: randomly initialized neural networks already contain small "winning ticket" subnetworks that can match full network performance when trained in isolation from their original initialization.
The Core Idea
Imagine buying 1,000 lottery tickets at once — statistically, one of them is more likely to win than if you'd bought just one. The Lottery Ticket Hypothesis proposes something similar happens in neural networks at initialization.
When you randomly initialize a large neural network, certain sparse subnetworks are already configured in a way that makes them capable of training to full accuracy. These "winning tickets" aren't found after training — they existed from the start, hidden within the random initialization.
How It Works
The hypothesis suggests:
- Train a dense network to completion
- Apply magnitude-based pruning to identify important weights
- Reset the remaining weights to their original initialization values
- Retrain this sparse subnetwork from scratch
Surprisingly, this sparse network (10-20% of original size) can match or even exceed the original network's accuracy — but only when using the original "lucky" initialization.
Why It Matters for Pruning
This finding challenges conventional wisdom. Previously, we thought pruning worked because training created important weights. The Lottery Ticket Hypothesis suggests those weights were always destined to be important due to their initialization. The dense network simply provided the resources to find them.
This has profound implications: instead of training massive networks and pruning afterward, perhaps we could identify winning tickets early and train only those sparse subnetworks from the start.
Key Takeaway: The Lottery Ticket Hypothesis reveals that sparse, trainable subnetworks exist at random initialization within larger networks, suggesting network overparameterization helps us find good initializations rather than creating them during training.