The SARSA Update Rule
What you'll learn: How SARSA updates Q-values using the actual action your policy takes next, not just the best possible action.
The Core Difference from Q-Learning
Remember Q-Learning's update rule uses the maximum Q-value of the next state—it's optimistic about what you could do. SARSA (State-Action-Reward-State-Action) is more cautious: it uses the Q-value of the action you actually take next according to your current policy.
The SARSA Update Equation
The SARSA update rule is:
Q(s, a) ← Q(s, a) + α[r + γQ(s', a') - Q(s, a)]
Where:
s, a: current state and actionr: reward receiveds', a': next state and the actual next action taken (sampled from your policy)α: learning rateγ: discount factor
The key insight: a' comes from following your policy (often ε-greedy), not from taking the maximum over all actions.
Why This Matters
Think of it like learning to drive. Q-Learning assumes you'll always make the perfect next move when estimating value. SARSA is realistic—it learns based on what you actually did, including occasional random exploration moves.
This makes SARSA on-policy: it learns the value of the policy it's currently following, including its exploration behavior. Q-Learning is off-policy: it learns the optimal policy regardless of what actions it explores with.
Key Takeaway: The SARSA update rule uses Q(s', a') where a' is the actual next action taken by the current policy, making it an on-policy algorithm that learns conservative, safer policies during exploration.