The Cliff Walking Problem
What you'll learn: You'll see how Q-learning's optimistic off-policy updates differ from SARSA's cautious on-policy behavior using a classic gridworld benchmark.
The Setup
The Cliff Walking problem is a standard gridworld where an agent starts at the bottom-left corner and must reach a goal at the bottom-right. However, the entire bottom row between start and goal is a dangerous "cliff"—stepping on it gives a large negative reward (-100) and resets the agent to start. Other moves give -1 reward. The agent can move up, down, left, or right.
Why This Problem Matters
This environment beautifully reveals the behavioral difference between Q-learning and SARSA:
Q-Learning: The Daredevil
Since Q-learning is off-policy, it learns the optimal policy (the shortest path hugging the cliff edge) regardless of which actions it actually takes during training. It updates Q-values assuming future actions will be greedy, even if exploration occasionally falls off the cliff. Result: learns the risky but optimal path.
SARSA: The Cautious Learner
SARSA is on-policy, meaning it learns the value of the policy it's currently following (including exploration). Since epsilon-greedy exploration occasionally causes cliff falls, SARSA learns that staying near the cliff is dangerous under its current exploratory behavior. Result: learns a safer path that stays farther from the cliff, trading optimality for safety.
The Insight
During training with ε-greedy exploration, Q-learning's final policy often walks the cliff edge (optimal but dangerous during learning), while SARSA's final policy takes the longer safe route. This demonstrates how off-policy methods can separate learning from behavior, while on-policy methods must account for exploration risks.
Key Takeaway: The Cliff Walking problem illustrates that Q-learning learns optimal policies despite exploratory behavior, while SARSA learns policies that are safe under exploration—a fundamental trade-off in temporal difference control methods.