Symmetry Breaking
What you'll learn: Why initializing all neural network weights to the same value blocks learning, and how randomness solves this problem.
The Problem: Identical Weights Create Identical Neurons
Imagine you're training a neural network with multiple neurons in a hidden layer. If you initialize all weights to the same value (say, all zeros or all ones), something disastrous happens: every neuron in that layer computes exactly the same output.
During backpropagation, these identical neurons also receive identical gradients. When you update the weights, all neurons update by the exact same amount. They remain clones of each other forever—no matter how long you train.
This is called the symmetry problem: neurons are symmetrically identical and cannot specialize to detect different features in your data.
Breaking the Symmetry with Randomness
The solution is elegantly simple: initialize weights randomly. Even small random differences between neurons create different initial outputs. During training, these differences amplify—each neuron develops its own specialty, learning to detect different patterns.
Think of it like this: if identical twins wore the same clothes, attended the same classes, and made identical choices every day, they'd never develop unique personalities. Random experiences (different weight initializations) allow them to become distinct individuals (specialized feature detectors).
Why This Matters
Without symmetry breaking:
- All neurons in a layer remain redundant copies
- The network cannot learn complex, diverse representations
- You effectively have one neuron repeated many times instead of many independent learners
Random initialization ensures each neuron starts on a slightly different path, enabling the rich, distributed representations that make deep learning powerful.
Key Takeaway: Initializing all weights to identical values prevents neurons from learning different features; random initialization breaks this symmetry and allows each neuron to specialize during training.