Course contentsShow
← Machine Learning and Deep Learning
Lesson 2207 of 3,53848. Deep Reinforcement Learning: Value-BasedFree lesson

From Q-Learning to Deep Q-Networks

Why tabular Q-learning fails in high-dimensional spaces and how neural networks approximate Q-functions.

From Q-Learning to Deep Q-Networks

What you'll learn: Why tabular Q-learning breaks down in complex environments and how neural networks come to the rescue.

The Scalability Problem

You've learned Q-learning stores Q-values in a table: one entry for every state-action pair. This works beautifully in simple environments like gridworlds with maybe hundreds of states. But what happens when you want an agent to play Atari games from raw pixels?

The curse of dimensionality strikes hard. An 84×84 grayscale screen has 256^(84×84) ≈ 10^14,000 possible states—more than atoms in the universe! You simply cannot build a table that large, let alone visit every state enough times to learn accurate Q-values.

Enter Function Approximation

Instead of memorizing Q-values for every state-action pair, we need a function approximator that can generalize: a model that learns patterns and estimates Q-values for states it's never seen before.

Think of it like learning to catch a ball. You don't memorize the trajectory of every possible throw—you learn the underlying physics and apply it to new situations.

Neural networks are perfect for this. They excel at finding patterns in high-dimensional data (like images) and can approximate complex functions. A Deep Q-Network (DQN) takes a state (say, game pixels) as input and outputs Q-values for all actions simultaneously.

The Core Idea

  • Tabular Q-learning: Q_table[state, action] = value
  • Deep Q-Network: Q(state, action) = neural_network(state)[action]

The network learns to map states to Q-values by minimizing the difference between predicted Q-values and target Q-values (just like the Q-learning update rule, but now updating network weights instead of table entries).

Key Takeaway: Deep Q-Networks replace the Q-table with a neural network, enabling Q-learning to scale from toy problems to high-dimensional environments like video games and robotics.