Course contentsShow
← Machine Learning and Deep Learning
Lesson 2175 of 3,53847. Reinforcement Learning: Temporal Difference MethodsFree lesson

The Q-Learning Update Rule

Understanding the Q-learning update with max operator and why it converges to optimal policy.

The Q-Learning Update Rule

What you'll learn: How Q-learning updates Q-values using the max operator to gradually converge toward the optimal policy.

The Core Update Formula

Q-learning uses a simple but powerful update rule that combines current knowledge with new experience:

Q(s, a) ← Q(s, a) + α[r + γ max Q(s', a') - Q(s, a)]
                              a'

Let's break this down in plain English.

Understanding Each Component

Q(s, a) is your current estimate of taking action a in state s.

α (alpha) is the learning rate—how much you trust new information vs. old estimates.

r is the immediate reward you just received.

γ (gamma) is the discount factor from earlier lessons—how much future rewards matter.

max Q(s', a') is the maximum Q-value possible in the next state s'—this is the key insight!

Why the Max Operator Matters

The max operator means you're always assuming you'll act optimally from the next state onward. Even if you're currently exploring randomly (off-policy behavior), your Q-value updates assume future perfection. This is what allows Q-learning to learn the optimal policy without actually following it during training.

Think of it like learning chess: you might make random experimental moves now, but when evaluating positions, you calculate assuming best play from that point forward.

Convergence to Optimality

Given enough exploration and appropriate learning rate decay, this update rule provably converges to Q*, the optimal action-value function. Once you have Q*, extracting the optimal policy is trivial: just pick argmax Q(s, a) for each state.

Key Takeaway: The Q-learning update rule uses max Q(s', a') to bootstrap toward optimal values regardless of the policy being followed, enabling off-policy learning that converges to the optimal policy with sufficient exploration.