Sample-Average Method
What you'll learn: How to estimate action values by simply averaging all the rewards you've observed for each action—the most intuitive approach to learning from experience.
The Core Idea
When you're trying to figure out how good an action is (its action value), the most straightforward approach is to try it multiple times and average the rewards you receive. This is the sample-average method.
If you've taken action a a total of N(a) times and received rewards R₁, R₂, ..., Rₙ, then your estimate of that action's value is:
Q(a) = (R₁ + R₂ + ... + Rₙ) / N(a)
By the law of large numbers, as you try an action more and more, this average converges to the true expected reward.
The Incremental Update Rule
Computing the full average every time is wasteful. Instead, we can update our estimate incrementally after each new reward:
Q_new(a) = Q_old(a) + (1/N(a)) × [R - Q_old(a)]
This says: adjust your current estimate toward the new observation, with the step size being 1/N(a). The term [R - Q_old(a)] is the error—how much the new reward differs from your current estimate.
Think of it like updating your opinion of a restaurant: if you've been there 10 times and think it's "7/10 quality," but your 11th visit feels like a "9," you nudge your overall rating up slightly (by 2/11 in this case).
Why It Matters
The sample-average method is the foundation for how agents learn action values in bandit problems and beyond. The incremental form reveals a pattern you'll see throughout RL: NewEstimate = OldEstimate + StepSize × Error.
Key Takeaway: The sample-average method estimates action values by averaging observed rewards, and can be computed incrementally using a step size of 1/N(a) to efficiently update estimates without storing all past rewards.