Course contentsShow
← Machine Learning and Deep Learning
Lesson 2199 of 3,53847. Reinforcement Learning: Temporal Difference MethodsFree lesson

Sample-Average Method

Computing action values by averaging observed rewards and understanding incremental update rules.

Sample-Average Method

What you'll learn: How to estimate action values by simply averaging all the rewards you've observed for each action—the most intuitive approach to learning from experience.

The Core Idea

When you're trying to figure out how good an action is (its action value), the most straightforward approach is to try it multiple times and average the rewards you receive. This is the sample-average method.

If you've taken action a a total of N(a) times and received rewards R₁, R₂, ..., Rₙ, then your estimate of that action's value is:

Q(a) = (R₁ + R₂ + ... + Rₙ) / N(a)

By the law of large numbers, as you try an action more and more, this average converges to the true expected reward.

The Incremental Update Rule

Computing the full average every time is wasteful. Instead, we can update our estimate incrementally after each new reward:

Q_new(a) = Q_old(a) + (1/N(a)) × [R - Q_old(a)]

This says: adjust your current estimate toward the new observation, with the step size being 1/N(a). The term [R - Q_old(a)] is the error—how much the new reward differs from your current estimate.

Think of it like updating your opinion of a restaurant: if you've been there 10 times and think it's "7/10 quality," but your 11th visit feels like a "9," you nudge your overall rating up slightly (by 2/11 in this case).

Why It Matters

The sample-average method is the foundation for how agents learn action values in bandit problems and beyond. The incremental form reveals a pattern you'll see throughout RL: NewEstimate = OldEstimate + StepSize × Error.

Key Takeaway: The sample-average method estimates action values by averaging observed rewards, and can be computed incrementally using a step size of 1/N(a) to efficiently update estimates without storing all past rewards.