From Q-Learning to Deep Q-Networks
What you'll learn: Why tabular Q-learning breaks down in complex environments and how neural networks come to the rescue.
The Scalability Problem
You've learned Q-learning stores Q-values in a table: one entry for every state-action pair. This works beautifully in simple environments like gridworlds with maybe hundreds of states. But what happens when you want an agent to play Atari games from raw pixels?
The curse of dimensionality strikes hard. An 84×84 grayscale screen has 256^(84×84) ≈ 10^14,000 possible states—more than atoms in the universe! You simply cannot build a table that large, let alone visit every state enough times to learn accurate Q-values.
Enter Function Approximation
Instead of memorizing Q-values for every state-action pair, we need a function approximator that can generalize: a model that learns patterns and estimates Q-values for states it's never seen before.
Think of it like learning to catch a ball. You don't memorize the trajectory of every possible throw—you learn the underlying physics and apply it to new situations.
Neural networks are perfect for this. They excel at finding patterns in high-dimensional data (like images) and can approximate complex functions. A Deep Q-Network (DQN) takes a state (say, game pixels) as input and outputs Q-values for all actions simultaneously.
The Core Idea
- Tabular Q-learning:
Q_table[state, action] = value - Deep Q-Network:
Q(state, action) = neural_network(state)[action]
The network learns to map states to Q-values by minimizing the difference between predicted Q-values and target Q-values (just like the Q-learning update rule, but now updating network weights instead of table entries).
Key Takeaway: Deep Q-Networks replace the Q-table with a neural network, enabling Q-learning to scale from toy problems to high-dimensional environments like video games and robotics.