Count-Based Exploration Bonuses
What You'll Learn
You'll learn how to add intrinsic rewards based on how often states are visited, encouraging agents to explore rarely-seen parts of their environment.
The Core Idea
Imagine you're exploring a new city. Naturally, you'd prefer visiting streets you haven't seen yet rather than walking down the same familiar road for the tenth time. Count-based exploration bonuses apply this intuition to reinforcement learning.
The idea is simple: keep track of how many times your agent has visited each state. Then, add a small "bonus reward" whenever the agent visits a rarely-seen state. This intrinsic reward encourages curiosity and systematic exploration, especially useful in sparse-reward environments where the agent might otherwise get stuck repeating familiar but unproductive behaviors.
How It Works
You maintain a count N(s) for each state s—how many times you've been there. The exploration bonus is inversely proportional to this count. A common formula is:
bonus(s) = β / √N(s)
where β is a scaling factor that controls exploration intensity.
You add this bonus to the environment's extrinsic reward:
total_reward = environment_reward + bonus(s)
States visited fewer times give larger bonuses, making them more attractive to explore. As a state is visited repeatedly, N(s) increases and the bonus shrinks, naturally shifting focus to other under-explored areas.
Building on What You Know
This complements strategies like epsilon-greedy and optimistic initialization. While those inject randomness or initial optimism, count-based bonuses provide a systematic, reward-driven mechanism that directly targets under-explored states.
Key Takeaway: Count-based exploration bonuses add intrinsic rewards inversely proportional to state visitation counts, systematically driving agents toward rarely-seen states and improving exploration efficiency.