Course contentsShow
← Machine Learning and Deep Learning
Lesson 2184 of 3,53847. Reinforcement Learning: Temporal Difference MethodsFree lesson

Implementing SARSA in Python

Coding a SARSA agent and comparing its learned behavior with Q-learning on the same task.

Implementing SARSA in Python

What you'll learn: You'll build a working SARSA agent in Python and see how its on-policy learning produces different behavior than Q-learning's off-policy approach.

The Core Idea

SARSA (State-Action-Reward-State-Action) is an on-policy TD control algorithm. Unlike Q-learning, which always uses the maximum Q-value for updates, SARSA updates based on the action it actually takes next. This makes SARSA more cautious—it learns the value of the policy it's currently following, including exploration randomness.

Think of it like this: Q-learning is an optimist who assumes you'll always make perfect future choices. SARSA is a realist who accounts for the fact that you sometimes explore randomly.

Building a SARSA Agent

Here's a minimal SARSA implementation:

import numpy as np

class SARSAAgent:
    def __init__(self, n_states, n_actions, alpha=0.1, gamma=0.99, epsilon=0.1):
        self.Q = np.zeros((n_states, n_actions))
        self.alpha = alpha
        self.gamma = gamma
        self.epsilon = epsilon
    
    def choose_action(self, state):
        if np.random.random() < self.epsilon:
            return np.random.randint(self.Q.shape[1])
        return np.argmax(self.Q[state])
    
    def update(self, state, action, reward, next_state, next_action):
        # SARSA uses the ACTUAL next action taken
        td_target = reward + self.gamma * self.Q[next_state, next_action]
        td_error = td_target - self.Q[state, action]
        self.Q[state, action] += self.alpha * td_error

The key difference: next_action comes from your epsilon-greedy policy, not max(Q[next_state]).

SARSA vs Q-Learning on Cliff Walking

On the Cliff Walking problem, SARSA learns a safer path away from the cliff edge because it accounts for random exploratory steps. Q-learning learns the optimal (risky) path right along the cliff, but suffers during training when epsilon-greedy makes it fall off.

Key Takeaway: SARSA's on-policy nature makes it learn conservative, safe policies during exploration, while Q-learning learns the optimal policy assuming perfect future execution—implement both to see how update rules shape learned behavior.