Course contentsShow
← Machine Learning and Deep Learning
Lesson 2285 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

Entropy Regularization for Exploration

Add policy entropy to the loss function to encourage exploration and prevent premature convergence.

Entropy Regularization for Exploration

What you'll learn: How to add an entropy bonus to your policy loss to maintain exploration and avoid getting stuck in suboptimal behaviors.


The Core Idea

In actor-critic methods, your policy network can sometimes become too confident too quickly—collapsing to always choosing one action with probability near 1.0 while ignoring others with near-zero probability. This premature convergence kills exploration and may trap you in local optima.

Entropy regularization solves this by adding a bonus term that rewards the policy for staying "uncertain" or "spread out" across multiple actions. Higher entropy means the policy distributes probability more evenly; lower entropy means it's more deterministic.

How It Works

Recall that entropy measures the randomness of a probability distribution. For a discrete policy with action probabilities π(a|s):

Entropy = -Σ π(a|s) log π(a|s)

In continuous spaces (Gaussian policies), entropy has a closed form involving the variance.

You add this entropy term to your policy loss (or equivalently, subtract it since we maximize):

Total Loss = Policy Gradient Loss - β × Entropy

The hyperparameter β (entropy coefficient) controls the exploration strength. Higher β → more exploration; lower β → more exploitation.

Why This Matters

Think of entropy regularization like penalizing overconfidence. Without it, your agent might learn "press button A" with 99.9% confidence early on, never discovering that "button B in certain states" leads to higher rewards. The entropy bonus keeps multiple options viable longer, encouraging thorough exploration before settling on a final strategy.

Key Takeaway:
Entropy regularization adds a policy entropy bonus to the loss function, encouraging the agent to maintain broad action distributions and explore thoroughly before converging to deterministic behavior.