Course contentsShow
← Machine Learning and Deep Learning
Lesson 2179 of 3,53847. Reinforcement Learning: Temporal Difference MethodsFree lesson

The Cliff Walking Problem

Illustrating behavioral differences between Q-learning and SARSA on a classic benchmark task.

The Cliff Walking Problem

What you'll learn: You'll see how Q-learning's optimistic off-policy updates differ from SARSA's cautious on-policy behavior using a classic gridworld benchmark.

The Setup

The Cliff Walking problem is a standard gridworld where an agent starts at the bottom-left corner and must reach a goal at the bottom-right. However, the entire bottom row between start and goal is a dangerous "cliff"—stepping on it gives a large negative reward (-100) and resets the agent to start. Other moves give -1 reward. The agent can move up, down, left, or right.

Why This Problem Matters

This environment beautifully reveals the behavioral difference between Q-learning and SARSA:

Q-Learning: The Daredevil

Since Q-learning is off-policy, it learns the optimal policy (the shortest path hugging the cliff edge) regardless of which actions it actually takes during training. It updates Q-values assuming future actions will be greedy, even if exploration occasionally falls off the cliff. Result: learns the risky but optimal path.

SARSA: The Cautious Learner

SARSA is on-policy, meaning it learns the value of the policy it's currently following (including exploration). Since epsilon-greedy exploration occasionally causes cliff falls, SARSA learns that staying near the cliff is dangerous under its current exploratory behavior. Result: learns a safer path that stays farther from the cliff, trading optimality for safety.

The Insight

During training with ε-greedy exploration, Q-learning's final policy often walks the cliff edge (optimal but dangerous during learning), while SARSA's final policy takes the longer safe route. This demonstrates how off-policy methods can separate learning from behavior, while on-policy methods must account for exploration risks.

Key Takeaway: The Cliff Walking problem illustrates that Q-learning learns optimal policies despite exploratory behavior, while SARSA learns policies that are safe under exploration—a fundamental trade-off in temporal difference control methods.