DQN Architecture and Components
What You'll Learn
You'll understand how a Deep Q-Network processes raw observations into Q-values through a neural architecture designed for reinforcement learning.
The Core Architecture
A Deep Q-Network (DQN) replaces the Q-table from Q-Learning with a neural network that approximates the Q-function. Think of it as a sophisticated pattern recognizer that learns to evaluate how good each action is in any given state.
Input Preprocessing
The DQN begins with input preprocessing that converts raw observations into a format the network can understand. For Atari games (where DQN was famously introduced), this means:
- Grayscale conversion
- Frame resizing (typically 84×84 pixels)
- Frame stacking (usually 4 consecutive frames) to capture motion
This preprocessing creates a rich state representation from pixels alone.
Convolutional Layers
Next come convolutional layers that extract spatial features from the input. These layers:
- Detect edges, textures, and patterns in early layers
- Learn game-specific features (e.g., enemy positions, obstacles) in deeper layers
- Gradually build hierarchical representations
The standard DQN uses 2-3 convolutional layers followed by fully connected layers, similar to image classification networks but tuned for sequential decision-making.
Q-Value Output Heads
Finally, the network produces Q-value outputs—one neuron per possible action. Unlike classification (which uses softmax), DQN outputs raw Q-values representing expected cumulative reward for each action. The agent then selects actions using epsilon-greedy or another exploration strategy based on these values.
Key Takeaway: The DQN architecture transforms raw observations through preprocessing, extracts spatial features via convolutional layers, and outputs a Q-value for each action—enabling deep learning to scale Q-Learning to high-dimensional state spaces like images.