Q-Learning vs SARSA: Key Differences
You'll learn: How off-policy Q-learning fundamentally differs from on-policy SARSA and when each algorithm shines.
The Core Distinction
The defining difference between Q-learning and SARSA lies in what action they learn from:
-
Q-learning (off-policy): Updates Q-values using the best possible next action (max Q-value), regardless of what action the agent actually takes next. It learns about the optimal policy while following an exploratory behavior policy.
-
SARSA (on-policy): Updates Q-values using the action the agent actually takes next, following its current policy. It learns about the policy it's executing.
Intuitive Analogy
Imagine learning to drive:
-
SARSA is like learning from your actual driving experience, including all your cautious decisions and mistakes. You learn to be a safe driver the way you actually drive.
-
Q-learning is like studying the optimal racing line in theory, even while you drive conservatively. You learn what the best driver would do, independent of your current caution level.
When to Use Each
Choose Q-learning when:
- You want to find the absolute optimal policy
- Your environment is deterministic or you can afford some risk during learning
- Exploration strategy is separate from the target policy
Choose SARSA when:
- Safety matters during learning (e.g., robotics, real-world systems)
- The environment has high stochasticity or "cliffs" where mistakes are costly
- You want convergence guarantees that match your actual behavior policy
Key Takeaway: Q-learning is off-policy and learns the optimal policy independently of behavior, making it more aggressive; SARSA is on-policy and learns the value of the policy it follows, making it more conservative and safer during training.