Course contentsShow
← Machine Learning and Deep Learning
Lesson 2183 of 3,53847. Reinforcement Learning: Temporal Difference MethodsFree lesson

Implementing Q-Learning in Python

Building a tabular Q-learning agent for a grid world environment with epsilon-greedy exploration.

Implementing Q-Learning in Python

What you'll learn: How to build a practical tabular Q-learning agent with epsilon-greedy exploration for a grid world environment.

The Core Idea

You've learned the Q-learning algorithm conceptually—now it's time to implement it. A tabular Q-learning agent stores Q-values in a table (dictionary or array) where each state-action pair has its own entry. Combined with epsilon-greedy exploration (choosing random actions with probability ε, greedy actions otherwise), this creates a complete learning system.

Think of it like learning chess moves: you keep a notebook (Q-table) recording how good each move is from each board position, occasionally trying random moves (exploration) to discover better strategies.

Building the Agent

Your implementation needs these components:

1. Q-Table Initialization
Create a data structure to store Q(s,a) values, typically initialized to zeros.

2. Epsilon-Greedy Action Selection

if random.random() < epsilon:
    action = random.choice(available_actions)  # Explore
else:
    action = argmax(Q[state])  # Exploit

3. Q-Learning Update Loop
For each step: take action, observe reward and next state, then update:

Q[state][action] += alpha * (reward + gamma * max(Q[next_state]) - Q[state][action])

4. Episode Management
Run multiple episodes, gradually decreasing epsilon to shift from exploration to exploitation.

Grid World Example Structure

In a grid world, states are grid positions, actions are {up, down, left, right}. Your agent starts at a position, takes actions, receives rewards (e.g., +1 for goal, -1 for cliff), and learns which actions lead to higher cumulative rewards.

Key Takeaway: A complete Q-learning implementation combines a Q-table data structure, epsilon-greedy exploration strategy, the Q-learning update rule, and episode-based training—all working together to learn optimal policies through interaction with the environment.