Noisy Networks for Exploration
What You'll Learn
You'll discover how adding learnable noise directly to neural network weights creates adaptive, state-dependent exploration that can replace epsilon-greedy strategies.
The Core Idea
Instead of using epsilon-greedy exploration (where you randomly select actions), Noisy Networks inject parametric noise directly into the network's weights. This means the network itself becomes stochastic, and the randomness in action selection naturally emerges from the noisy computations.
Think of it like this: epsilon-greedy is like flipping a coin before deciding whether to explore. Noisy Networks are like having slightly shaky hands that introduce randomness while you make the decision. The key breakthrough? The network learns how much shakiness (noise) is useful in different situations.
How It Works
In a standard DQN layer, you compute:
y = Wx + b
With Noisy Networks, you replace the fixed weights with noisy versions:
y = (μ_w + σ_w ⊙ ε_w)x + (μ_b + σ_b ⊙ ε_b)
Where:
μ_wandμ_bare learnable mean parametersσ_wandσ_bare learnable standard deviation parametersε_wandε_bare random noise samples (e.g., from a Gaussian)⊙represents element-wise multiplication
The beauty? During training, backpropagation adjusts both the means and the noise magnitudes. Early in training, the network keeps high noise for exploration. As it learns, it can reduce noise in well-understood states while maintaining it elsewhere—achieving state-dependent exploration automatically.
Why It Matters
No more epsilon scheduling or manual exploration tuning. The agent learns when and how much to explore based on the task itself.
Key Takeaway: Noisy Networks replace epsilon-greedy with learnable parametric noise in network weights, enabling automatic state-dependent exploration that improves as the agent learns.