Course contentsShow
← Machine Learning and Deep Learning
Lesson 2145 of 3,53846. Reinforcement Learning: FundamentalsFree lesson

Gridworld: A Classic MDP Example

Walking through a simple gridworld problem to illustrate states, actions, rewards, and transitions.

Gridworld: A Classic MDP Example

What you'll learn: How a simple grid-based navigation problem demonstrates all the core components of a Markov Decision Process in action.

The Classic Gridworld Setup

Imagine a robot navigating a simple grid—like a chessboard. This "gridworld" is the canonical toy problem for understanding MDPs because it makes abstract concepts concrete and visual.

The typical setup:

  • States: Each cell in the grid is a state (e.g., position (2,3))
  • Actions: Move up, down, left, or right
  • Transitions: Actions deterministically or stochastically move the agent to adjacent cells (hitting walls keeps you in place)
  • Rewards: Most cells give -1 (encouraging efficiency), a goal cell gives +10, a trap cell gives -10
  • Episodes: Start at a designated cell, end when reaching goal or trap

Why Gridworld Matters

This simple environment illustrates every MDP concept you've learned:

  • The Markov property holds: knowing your current grid position tells you everything needed for decision-making
  • You can visualize policies as arrows showing which direction to move from each cell
  • Value functions V(s) assign a number to each cell representing expected future reward
  • The discount factor γ determines whether the agent rushes toward the goal or takes safer, longer paths
  • Q-functions Q(s,a) let you compare "move up from (2,3)" versus "move right from (2,3)"

Think of gridworld as a video game level where you're learning the optimal strategy through trial and error—except we can mathematically analyze every possibility.

Visualizing the Solution

An optimal policy might show: from each cell, which action maximizes expected return. Near the goal, all arrows point toward it. Near traps, arrows route around them. This visual intuition builds the foundation for more complex reinforcement learning environments.

Key Takeaway: Gridworld transforms abstract MDP components—states, actions, transitions, rewards, policies, and value functions—into a simple, visual navigation problem that you can understand at a glance.