Course contentsShow
← Machine Learning and Deep Learning
Lesson 2176 of 3,53847. Reinforcement Learning: Temporal Difference MethodsFree lesson

SARSA: On-Policy TD Control

Introducing SARSA (State-Action-Reward-State-Action) as an on-policy alternative to Q-learning.

SARSA: On-Policy TD Control

What you'll learn: You'll discover SARSA, a temporal difference algorithm that learns from the actions it actually takes, not hypothetical optimal ones.

The Core Idea

SARSA stands for State-Action-Reward-State-Action, describing the sequence of information it uses for learning. Unlike Q-learning (which you've just learned), SARSA is an on-policy algorithm—it learns the value of the policy it's currently following, including its exploratory actions.

Think of it this way: Q-learning is like a driving student who always assumes they'll make perfect decisions in the future, while SARSA is more realistic—it learns based on the actual (sometimes exploratory) decisions it makes.

How SARSA Works

SARSA follows this pattern:

  1. You're in state S
  2. You take action A (according to your current policy, e.g., ε-greedy)
  3. You receive reward R
  4. You land in state S'
  5. You select the next action A' (again, using your current policy)

The key difference from Q-learning: SARSA uses the actual next action A' it will take (chosen by its exploration policy), not the greedy maximum action.

The Update Rule

While Q-learning updates with the max over all actions, SARSA updates with the action it actually selected:

Q(S, A) ← Q(S, A) + α[R + γQ(S', A') - Q(S, A)]

Notice Q(S', A') uses the chosen next action A', not max Q(S', a). This makes SARSA more conservative when exploring—it accounts for the fact that it might take suboptimal exploratory actions.

Key Takeaway: SARSA is an on-policy TD control algorithm that learns the value of the policy it's actually following (including exploration), making it more conservative but realistic compared to Q-learning's off-policy optimistic approach.