Course contentsShow
← Machine Learning and Deep Learning
Lesson 2208 of 3,53848. Deep Reinforcement Learning: Value-BasedFree lesson

DQN Architecture and Components

The standard DQN network architecture: input preprocessing, convolutional layers, and Q-value output heads.

DQN Architecture and Components

What You'll Learn

You'll understand how a Deep Q-Network processes raw observations into Q-values through a neural architecture designed for reinforcement learning.

The Core Architecture

A Deep Q-Network (DQN) replaces the Q-table from Q-Learning with a neural network that approximates the Q-function. Think of it as a sophisticated pattern recognizer that learns to evaluate how good each action is in any given state.

Input Preprocessing

The DQN begins with input preprocessing that converts raw observations into a format the network can understand. For Atari games (where DQN was famously introduced), this means:

  • Grayscale conversion
  • Frame resizing (typically 84×84 pixels)
  • Frame stacking (usually 4 consecutive frames) to capture motion

This preprocessing creates a rich state representation from pixels alone.

Convolutional Layers

Next come convolutional layers that extract spatial features from the input. These layers:

  • Detect edges, textures, and patterns in early layers
  • Learn game-specific features (e.g., enemy positions, obstacles) in deeper layers
  • Gradually build hierarchical representations

The standard DQN uses 2-3 convolutional layers followed by fully connected layers, similar to image classification networks but tuned for sequential decision-making.

Q-Value Output Heads

Finally, the network produces Q-value outputs—one neuron per possible action. Unlike classification (which uses softmax), DQN outputs raw Q-values representing expected cumulative reward for each action. The agent then selects actions using epsilon-greedy or another exploration strategy based on these values.

Key Takeaway: The DQN architecture transforms raw observations through preprocessing, extracts spatial features via convolutional layers, and outputs a Q-value for each action—enabling deep learning to scale Q-Learning to high-dimensional state spaces like images.