Course contentsShow
← Machine Learning and Deep Learning
Lesson 2202 of 3,53847. Reinforcement Learning: Temporal Difference MethodsFree lesson

Upper Confidence Bound (UCB) Selection

Selecting actions based on potential optimality using uncertainty estimates and confidence intervals.

Upper Confidence Bound (UCB) Selection

You'll learn how UCB balances exploration and exploitation by selecting actions that could be optimal based on uncertainty.

The Core Idea

Upper Confidence Bound (UCB) selection is a smarter exploration strategy than epsilon-greedy. Instead of randomly exploring, UCB selects actions based on two factors: their estimated value and how uncertain we are about that estimate.

Think of it like choosing restaurants: you might pick one you haven't tried much (high uncertainty) even if a familiar place has slightly better average reviews. The untried restaurant could be amazing—you just don't know yet.

How UCB Works

UCB assigns each action an "optimistic" score by adding an uncertainty bonus to its estimated value. Actions you've tried less get a bigger bonus, encouraging you to explore them. As you gather more samples, the uncertainty shrinks, and the bonus decreases.

The formula (you've seen it in the previous lesson) typically looks like:

UCB_score = Q(a) + c × sqrt(ln(t) / N(a))

Where:

  • Q(a) is your current value estimate for action a
  • N(a) is how many times you've tried action a
  • t is the total number of actions taken
  • c is a tunable exploration parameter

The square root term grows as actions become less-tried relative to others, creating automatic exploration pressure where it's needed most.

Why It's Better

UCB systematically explores promising uncertainty rather than randomly. It naturally shifts from exploration to exploitation as confidence grows, without needing manually tuned epsilon schedules.

Key Takeaway: UCB selects actions by optimistically assuming uncertain actions might be the best, using confidence intervals to balance between exploiting known good actions and exploring uncertain ones.