Course contentsShow
← Machine Learning and Deep Learning
Lesson 2198 of 3,53847. Reinforcement Learning: Temporal Difference MethodsFree lesson

Action-Value Functions in Bandits

Defining Q(a) as expected reward for action a and why we estimate rather than know these values.

Action-Value Functions in Bandits

What you'll learn: How we define and estimate the expected reward for each action in a multi-armed bandit problem.

The Core Idea

In the multi-armed bandit problem, each action (or "arm") has a true, underlying expected reward—but we don't know it. We call this expected reward Q(a), the action-value function for action a. It represents the average reward you'd get if you kept pulling that arm forever.

Think of it like choosing between restaurants: each restaurant has a "true" average quality score, but you haven't eaten there infinitely many times to know it precisely. You can only estimate based on your limited experiences.

Why Estimation is Necessary

We estimate rather than know Q(a) because:

  1. We don't observe the true distribution – In real problems, we don't have access to the exact probability distribution of rewards for each action
  2. We learn through interaction – We only discover information by taking actions and observing rewards
  3. Efficiency matters – We can't pull every arm infinitely to learn the exact expected value; we need to balance learning with earning rewards

The Estimation Process

As you select actions and receive rewards, you maintain an estimated action-value (often denoted Q̂(a) or just Q(a) in practice). A simple estimate is the sample average:

Estimated Q(a) = (sum of rewards from action a) / (number of times action a was taken)

This estimate improves as you gather more experience with each action, gradually converging toward the true Q(a).

Key Takeaway: Action-value functions Q(a) represent the true expected reward per action, but since we can't know them directly, we must estimate them from observed rewards—forming the foundation of how bandit algorithms learn and improve.