Weight Normalization
What you'll learn: How to stabilize training by reparameterizing weight vectors into separate magnitude and direction components.
The Core Idea
Most regularization techniques we've seen (L1, L2, Dropout, Batch Norm) modify activations or add penalties. Weight Normalization takes a different approach: it reparameterizes how we represent the weights themselves.
Instead of learning a weight vector w directly, Weight Normalization splits it into:
- Direction: a unit vector v (where to point)
- Magnitude: a scalar g (how far to go)
Mathematically: w = g · (v / ||v||)
Why This Helps
Think of steering a car: you control direction (where the wheel points) and speed (how hard you press the gas) separately. Weight Normalization does the same for learning.
By decoupling magnitude from direction:
- Faster convergence: The optimizer can adjust "how much" and "which way" independently
- Improved gradient flow: The reparameterization has better conditioning properties
- Simpler than Batch Norm: No dependence on batch statistics, works naturally with small batches or online learning
Unlike Batch/Layer Normalization which normalize activations, Weight Normalization normalizes the parameters themselves before they're even used in computation.
When to Use It
Weight Normalization shines in scenarios where batch statistics are unreliable (recurrent networks, reinforcement learning, very small batches). It's also simpler at inference—no moving averages to track like Batch Normalization requires.
Key Takeaway: Weight Normalization reparameterizes weight vectors into magnitude and direction, allowing the optimizer to learn each aspect independently for more stable and efficient training dynamics.