Course contentsShow
← Machine Learning and Deep Learning
Lesson 2232 of 3,53848. Deep Reinforcement Learning: Value-BasedFree lesson

Noisy Networks for Exploration

Parametric noise in network weights for state-dependent exploration that replaces epsilon-greedy.

Noisy Networks for Exploration

What You'll Learn

You'll discover how adding learnable noise directly to neural network weights creates adaptive, state-dependent exploration that can replace epsilon-greedy strategies.


The Core Idea

Instead of using epsilon-greedy exploration (where you randomly select actions), Noisy Networks inject parametric noise directly into the network's weights. This means the network itself becomes stochastic, and the randomness in action selection naturally emerges from the noisy computations.

Think of it like this: epsilon-greedy is like flipping a coin before deciding whether to explore. Noisy Networks are like having slightly shaky hands that introduce randomness while you make the decision. The key breakthrough? The network learns how much shakiness (noise) is useful in different situations.

How It Works

In a standard DQN layer, you compute:

y = Wx + b

With Noisy Networks, you replace the fixed weights with noisy versions:

y = (μ_w + σ_w ⊙ ε_w)x + (μ_b + σ_b ⊙ ε_b)

Where:

  • μ_w and μ_b are learnable mean parameters
  • σ_w and σ_b are learnable standard deviation parameters
  • ε_w and ε_b are random noise samples (e.g., from a Gaussian)
  • ⊙ represents element-wise multiplication

The beauty? During training, backpropagation adjusts both the means and the noise magnitudes. Early in training, the network keeps high noise for exploration. As it learns, it can reduce noise in well-understood states while maintaining it elsewhere—achieving state-dependent exploration automatically.

Why It Matters

No more epsilon scheduling or manual exploration tuning. The agent learns when and how much to explore based on the task itself.

Key Takeaway: Noisy Networks replace epsilon-greedy with learnable parametric noise in network weights, enabling automatic state-dependent exploration that improves as the agent learns.