Implementing Q-Learning in Python
What you'll learn: How to build a practical tabular Q-learning agent with epsilon-greedy exploration for a grid world environment.
The Core Idea
You've learned the Q-learning algorithm conceptually—now it's time to implement it. A tabular Q-learning agent stores Q-values in a table (dictionary or array) where each state-action pair has its own entry. Combined with epsilon-greedy exploration (choosing random actions with probability ε, greedy actions otherwise), this creates a complete learning system.
Think of it like learning chess moves: you keep a notebook (Q-table) recording how good each move is from each board position, occasionally trying random moves (exploration) to discover better strategies.
Building the Agent
Your implementation needs these components:
1. Q-Table Initialization
Create a data structure to store Q(s,a) values, typically initialized to zeros.
2. Epsilon-Greedy Action Selection
if random.random() < epsilon:
action = random.choice(available_actions) # Explore
else:
action = argmax(Q[state]) # Exploit
3. Q-Learning Update Loop
For each step: take action, observe reward and next state, then update:
Q[state][action] += alpha * (reward + gamma * max(Q[next_state]) - Q[state][action])
4. Episode Management
Run multiple episodes, gradually decreasing epsilon to shift from exploration to exploitation.
Grid World Example Structure
In a grid world, states are grid positions, actions are {up, down, left, right}. Your agent starts at a position, takes actions, receives rewards (e.g., +1 for goal, -1 for cliff), and learns which actions lead to higher cumulative rewards.
Key Takeaway: A complete Q-learning implementation combines a Q-table data structure, epsilon-greedy exploration strategy, the Q-learning update rule, and episode-based training—all working together to learn optimal policies through interaction with the environment.