Course contentsShow
← Machine Learning and Deep Learning
Lesson 2174 of 3,53847. Reinforcement Learning: Temporal Difference MethodsFree lesson

Q-Learning: Off-Policy TD Control

Introducing Q-learning as an off-policy algorithm that learns the optimal action-value function.

Q-Learning: Off-Policy TD Control

What you'll learn: How Q-learning learns the optimal action-value function by breaking free from the policy it's currently following.

The Core Idea

Q-learning is a temporal difference method that learns the optimal Q-function (Q*) directly, without needing to follow the optimal policy while learning. This "off-policy" nature is its superpower: your agent can explore randomly or follow a safe policy while still learning what the best actions would be.

Think of it like watching a friend make mistakes playing a video game. Even though they're choosing suboptimal moves, you can still learn which actions would have been best. You're learning about the optimal strategy while observing a different one.

The Q-Learning Update Rule

The algorithm updates Q-values using this simple rule:

Q(s, a) ← Q(s, a) + α [r + γ max Q(s', a') - Q(s, a)]
                              a'

The key innovation: we use max over all actions in the next state, regardless of which action our policy actually took. This bootstraps toward the optimal Q* values.

Off-Policy Explained

  • Behavior policy: What we actually do (often ε-greedy for exploration)
  • Target policy: What we're learning about (the greedy/optimal policy)

Q-learning learns about the greedy target policy (argmax Q) while following an exploratory behavior policy. This decoupling enables safer exploration and faster learning from suboptimal experience.

Why It Matters

Unlike on-policy methods (like SARSA), Q-learning can learn from any experience—replayed data, demonstrations, or random exploration—making it more data-efficient and flexible.

Key Takeaway: Q-learning is an off-policy TD control algorithm that learns the optimal action-value function Q* by always bootstrapping toward the maximum Q-value in the next state, regardless of the action actually taken.