Course contentsShow
← Machine Learning and Deep Learning
Lesson 2194 of 3,53847. Reinforcement Learning: Temporal Difference MethodsFree lesson

Count-Based Exploration Bonuses

Adding intrinsic rewards based on visitation counts to encourage exploration of rarely-seen states.

Count-Based Exploration Bonuses

What You'll Learn

You'll learn how to add intrinsic rewards based on how often states are visited, encouraging agents to explore rarely-seen parts of their environment.

The Core Idea

Imagine you're exploring a new city. Naturally, you'd prefer visiting streets you haven't seen yet rather than walking down the same familiar road for the tenth time. Count-based exploration bonuses apply this intuition to reinforcement learning.

The idea is simple: keep track of how many times your agent has visited each state. Then, add a small "bonus reward" whenever the agent visits a rarely-seen state. This intrinsic reward encourages curiosity and systematic exploration, especially useful in sparse-reward environments where the agent might otherwise get stuck repeating familiar but unproductive behaviors.

How It Works

You maintain a count N(s) for each state s—how many times you've been there. The exploration bonus is inversely proportional to this count. A common formula is:

bonus(s) = β / √N(s)

where β is a scaling factor that controls exploration intensity.

You add this bonus to the environment's extrinsic reward:

total_reward = environment_reward + bonus(s)

States visited fewer times give larger bonuses, making them more attractive to explore. As a state is visited repeatedly, N(s) increases and the bonus shrinks, naturally shifting focus to other under-explored areas.

Building on What You Know

This complements strategies like epsilon-greedy and optimistic initialization. While those inject randomness or initial optimism, count-based bonuses provide a systematic, reward-driven mechanism that directly targets under-explored states.

Key Takeaway: Count-based exploration bonuses add intrinsic rewards inversely proportional to state visitation counts, systematically driving agents toward rarely-seen states and improving exploration efficiency.