Gridworld: A Classic MDP Example
What you'll learn: How a simple grid-based navigation problem demonstrates all the core components of a Markov Decision Process in action.
The Classic Gridworld Setup
Imagine a robot navigating a simple grid—like a chessboard. This "gridworld" is the canonical toy problem for understanding MDPs because it makes abstract concepts concrete and visual.
The typical setup:
- States: Each cell in the grid is a state (e.g., position (2,3))
- Actions: Move up, down, left, or right
- Transitions: Actions deterministically or stochastically move the agent to adjacent cells (hitting walls keeps you in place)
- Rewards: Most cells give -1 (encouraging efficiency), a goal cell gives +10, a trap cell gives -10
- Episodes: Start at a designated cell, end when reaching goal or trap
Why Gridworld Matters
This simple environment illustrates every MDP concept you've learned:
- The Markov property holds: knowing your current grid position tells you everything needed for decision-making
- You can visualize policies as arrows showing which direction to move from each cell
- Value functions V(s) assign a number to each cell representing expected future reward
- The discount factor γ determines whether the agent rushes toward the goal or takes safer, longer paths
- Q-functions Q(s,a) let you compare "move up from (2,3)" versus "move right from (2,3)"
Think of gridworld as a video game level where you're learning the optimal strategy through trial and error—except we can mathematically analyze every possibility.
Visualizing the Solution
An optimal policy might show: from each cell, which action maximizes expected return. Near the goal, all arrows point toward it. Near traps, arrows route around them. This visual intuition builds the foundation for more complex reinforcement learning environments.
Key Takeaway: Gridworld transforms abstract MDP components—states, actions, transitions, rewards, policies, and value functions—into a simple, visual navigation problem that you can understand at a glance.