Course contentsShow
← Machine Learning and Deep Learning
Lesson 2180 of 3,53847. Reinforcement Learning: Temporal Difference MethodsFree lesson

Expected SARSA

Introducing Expected SARSA that averages over actions instead of sampling a single next action.

Expected SARSA

What you'll learn: How Expected SARSA improves upon SARSA by averaging over all possible next actions instead of sampling just one.

The Core Idea

In standard SARSA, you pick an actual next action A' (often using ε-greedy) and update your Q-values based on Q(S', A'). But that single action is just one sample—it might be a random exploration move that doesn't reflect the true expected value of being in state S'.

Expected SARSA solves this by computing the expected Q-value across all possible actions in the next state, weighted by how likely your policy is to choose each action. Instead of relying on one sampled action, you average over the entire action distribution.

The Update Rule

The Expected SARSA update is:

Q(S, A) ← Q(S, A) + α[R + γ · E[Q(S', a)] - Q(S, A)]

Where:

E[Q(S', a)] = Σ π(a|S') · Q(S', a)

You sum over all actions a in state S', weighted by the probability π(a|S') that your policy would choose that action.

Why It Matters

Think of it like weather forecasting: SARSA looks at tomorrow's actual weather (one sample), while Expected SARSA considers the probability-weighted forecast for all possible weather conditions. The latter gives you a more stable, less noisy estimate.

Expected SARSA reduces variance in updates because it doesn't depend on the randomness of action selection—especially important when using ε-greedy policies where random actions can skew your estimates.

Key Takeaway: Expected SARSA averages Q-values over all possible next actions weighted by policy probabilities, providing more stable updates than standard SARSA while retaining the on-policy learning approach.