Course contentsShow
← Machine Learning and Deep Learning
Lesson 761 of 3,53818. Regularization Techniques for Deep NetworksFree lesson

Weight Normalization

Learn an alternative approach that reparameterizes weight vectors into magnitude and direction, decoupling learning dynamics.

Weight Normalization

What you'll learn: How to stabilize training by reparameterizing weight vectors into separate magnitude and direction components.

The Core Idea

Most regularization techniques we've seen (L1, L2, Dropout, Batch Norm) modify activations or add penalties. Weight Normalization takes a different approach: it reparameterizes how we represent the weights themselves.

Instead of learning a weight vector w directly, Weight Normalization splits it into:

  • Direction: a unit vector v (where to point)
  • Magnitude: a scalar g (how far to go)

Mathematically: w = g · (v / ||v||)

Why This Helps

Think of steering a car: you control direction (where the wheel points) and speed (how hard you press the gas) separately. Weight Normalization does the same for learning.

By decoupling magnitude from direction:

  1. Faster convergence: The optimizer can adjust "how much" and "which way" independently
  2. Improved gradient flow: The reparameterization has better conditioning properties
  3. Simpler than Batch Norm: No dependence on batch statistics, works naturally with small batches or online learning

Unlike Batch/Layer Normalization which normalize activations, Weight Normalization normalizes the parameters themselves before they're even used in computation.

When to Use It

Weight Normalization shines in scenarios where batch statistics are unreliable (recurrent networks, reinforcement learning, very small batches). It's also simpler at inference—no moving averages to track like Batch Normalization requires.

Key Takeaway: Weight Normalization reparameterizes weight vectors into magnitude and direction, allowing the optimizer to learn each aspect independently for more stable and efficient training dynamics.