Greedy Action Selection and Its Limitations
What you'll learn: How choosing the seemingly best action every time can trap you in suboptimal solutions by preventing the discovery of better alternatives.
The Core Idea
In reinforcement learning, greedy action selection means always picking the action with the highest estimated value according to your current Q-function or value estimates. At each state, you simply choose:
action = argmax(Q(state, a) for all actions a)
This sounds logical—why wouldn't you always choose what appears best? But there's a critical flaw.
The Trap of Pure Greed
Imagine you're in a new city looking for restaurants. The first place you try is pretty good (7/10). If you're purely greedy, you'll return there every single night because it's currently your "best known option." You'll never discover the amazing restaurant two blocks away that would've been a 10/10.
Greedy policies exploit current knowledge but never explore alternatives. Your Q-values are just estimates—often initialized randomly or arbitrarily. Early in learning, these estimates are wildly inaccurate. If you greedily commit to actions based on noisy initial estimates, you'll:
- Miss potentially superior actions that started with unlucky initial values
- Get stuck in local optima
- Fail to adapt when the environment changes
- Never gather enough experience to improve estimates for unexplored actions
When Greed Fails
In the Cliff Walking problem you've seen, a greedy learner might stumble upon a mediocre safe path early and never risk exploring shortcuts. The Q-values for unexplored actions remain frozen at their initial (often zero) values, forever seeming "worse" than the known path.
Key Takeaway: Pure greedy action selection exploits current estimates but completely abandons exploration, preventing the agent from discovering better actions and leading to consistently suboptimal policies.