SARSA: On-Policy TD Control
What you'll learn: You'll discover SARSA, a temporal difference algorithm that learns from the actions it actually takes, not hypothetical optimal ones.
The Core Idea
SARSA stands for State-Action-Reward-State-Action, describing the sequence of information it uses for learning. Unlike Q-learning (which you've just learned), SARSA is an on-policy algorithm—it learns the value of the policy it's currently following, including its exploratory actions.
Think of it this way: Q-learning is like a driving student who always assumes they'll make perfect decisions in the future, while SARSA is more realistic—it learns based on the actual (sometimes exploratory) decisions it makes.
How SARSA Works
SARSA follows this pattern:
- You're in state S
- You take action A (according to your current policy, e.g., ε-greedy)
- You receive reward R
- You land in state S'
- You select the next action A' (again, using your current policy)
The key difference from Q-learning: SARSA uses the actual next action A' it will take (chosen by its exploration policy), not the greedy maximum action.
The Update Rule
While Q-learning updates with the max over all actions, SARSA updates with the action it actually selected:
Q(S, A) ← Q(S, A) + α[R + γQ(S', A') - Q(S, A)]
Notice Q(S', A') uses the chosen next action A', not max Q(S', a). This makes SARSA more conservative when exploring—it accounts for the fact that it might take suboptimal exploratory actions.
Key Takeaway: SARSA is an on-policy TD control algorithm that learns the value of the policy it's actually following (including exploration), making it more conservative but realistic compared to Q-learning's off-policy optimistic approach.