Course contentsShow
← Machine Learning and Deep Learning
Lesson 666 of 3,53816. Activation Functions and Weight InitializationFree lesson

Symmetry Breaking

Why initializing all weights to the same value prevents learning and how randomness breaks symmetry.

Symmetry Breaking

What you'll learn: Why initializing all neural network weights to the same value blocks learning, and how randomness solves this problem.

The Problem: Identical Weights Create Identical Neurons

Imagine you're training a neural network with multiple neurons in a hidden layer. If you initialize all weights to the same value (say, all zeros or all ones), something disastrous happens: every neuron in that layer computes exactly the same output.

During backpropagation, these identical neurons also receive identical gradients. When you update the weights, all neurons update by the exact same amount. They remain clones of each other forever—no matter how long you train.

This is called the symmetry problem: neurons are symmetrically identical and cannot specialize to detect different features in your data.

Breaking the Symmetry with Randomness

The solution is elegantly simple: initialize weights randomly. Even small random differences between neurons create different initial outputs. During training, these differences amplify—each neuron develops its own specialty, learning to detect different patterns.

Think of it like this: if identical twins wore the same clothes, attended the same classes, and made identical choices every day, they'd never develop unique personalities. Random experiences (different weight initializations) allow them to become distinct individuals (specialized feature detectors).

Why This Matters

Without symmetry breaking:

  • All neurons in a layer remain redundant copies
  • The network cannot learn complex, diverse representations
  • You effectively have one neuron repeated many times instead of many independent learners

Random initialization ensures each neuron starts on a slightly different path, enabling the rich, distributed representations that make deep learning powerful.

Key Takeaway: Initializing all weights to identical values prevents neurons from learning different features; random initialization breaks this symmetry and allows each neuron to specialize during training.