Course contentsShow
← Machine Learning and Deep Learning
Lesson 2177 of 3,53847. Reinforcement Learning: Temporal Difference MethodsFree lesson

The SARSA Update Rule

Deriving the SARSA update equation and understanding how it uses the actual next action taken.

The SARSA Update Rule

What you'll learn: How SARSA updates Q-values using the actual action your policy takes next, not just the best possible action.

The Core Difference from Q-Learning

Remember Q-Learning's update rule uses the maximum Q-value of the next state—it's optimistic about what you could do. SARSA (State-Action-Reward-State-Action) is more cautious: it uses the Q-value of the action you actually take next according to your current policy.

The SARSA Update Equation

The SARSA update rule is:

Q(s, a) ← Q(s, a) + α[r + γQ(s', a') - Q(s, a)]

Where:

  • s, a: current state and action
  • r: reward received
  • s', a': next state and the actual next action taken (sampled from your policy)
  • α: learning rate
  • γ: discount factor

The key insight: a' comes from following your policy (often ε-greedy), not from taking the maximum over all actions.

Why This Matters

Think of it like learning to drive. Q-Learning assumes you'll always make the perfect next move when estimating value. SARSA is realistic—it learns based on what you actually did, including occasional random exploration moves.

This makes SARSA on-policy: it learns the value of the policy it's currently following, including its exploration behavior. Q-Learning is off-policy: it learns the optimal policy regardless of what actions it explores with.

Key Takeaway: The SARSA update rule uses Q(s', a') where a' is the actual next action taken by the current policy, making it an on-policy algorithm that learns conservative, safer policies during exploration.