Course contentsShow
← Machine Learning and Deep Learning
Lesson 2217 of 3,53848. Deep Reinforcement Learning: Value-BasedFree lesson

Handling Terminal States

Special treatment of episode termination in the Bellman equation and replay buffer implementation.

Handling Terminal States

What you'll learn: How to properly treat episode-ending states when computing Q-values and storing experiences in Deep Q-Networks.

Why Terminal States Need Special Treatment

In reinforcement learning, a terminal state is where an episode ends—the game is over, the agent reached the goal, or failed catastrophically. These states are fundamentally different because there's no "next state" to bootstrap from.

Think of it like a chess game: once you reach checkmate, there are no more moves. The value of that final position depends only on the immediate reward (win/loss), not on any future states—because there aren't any.

The Bellman Equation Problem

Recall the standard Q-learning update:

Q(s, a) ← Q(s, a) + α[r + γ * max Q(s', a') - Q(s, a)]

When s' is terminal, there's no max Q(s', a') to compute—no actions exist there. If you blindly apply the formula, you'd incorrectly bootstrap from garbage values.

The Solution: Zero Out Future Returns

For terminal states, the Bellman equation simplifies to:

Q(s, a) ← Q(s, a) + α[r - Q(s, a)]

The discount term vanishes: γ * max Q(s', a') = 0

Implementation in the Replay Buffer

When storing transitions, include a done flag:

# Store: (state, action, reward, next_state, done)
buffer.append((s, a, r, s_next, done))

# During training:
if done:
    target = reward
else:
    target = reward + gamma * max_next_Q

This boolean flag ensures your DQN learns that terminal states have no future value—only the immediate outcome matters.

Key Takeaway: Always treat terminal states specially by zeroing out the bootstrap term in the Bellman equation and using a done flag in your replay buffer to prevent learning incorrect Q-values from non-existent future states.