Course contentsShow
← Machine Learning and Deep Learning
Lesson 2239 of 3,53848. Deep Reinforcement Learning: Value-BasedFree lesson

Designing the Q-Network in PyTorch

Create a neural network for Q-value approximation with appropriate input/output dimensions and hidden layer architecture.

Designing the Q-Network in PyTorch

You'll learn how to build a neural network that maps environment states to Q-values for all possible actions.

The Core Idea

In Deep Q-Networks (DQN), we replace the Q-table with a neural network that approximates Q-values. Instead of storing values for every state-action pair (impossible in large or continuous spaces), the Q-network takes a state as input and outputs Q-values for every action simultaneously. Think of it as a function that compresses infinite possibilities into a manageable prediction model.

Network Architecture Fundamentals

Your Q-network needs:

  1. Input layer: Dimension matches your state representation (e.g., 4 values for CartPole: position, velocity, angle, angular velocity)
  2. Hidden layers: Typically 1-3 fully connected layers with ReLU activation to learn non-linear patterns
  3. Output layer: One neuron per action (e.g., 2 outputs for "left" and "right" in CartPole)

PyTorch Implementation

import torch
import torch.nn as nn

class QNetwork(nn.Module):
    def __init__(self, state_dim, action_dim, hidden_dim=128):
        super(QNetwork, self).__init__()
        
        self.network = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, action_dim)
        )
    
    def forward(self, state):
        return self.network(state)

This simple architecture takes state vectors, processes them through two hidden layers (128 neurons each), and outputs Q-values for each action—no softmax needed since we're predicting values, not probabilities.

Key Takeaway: The Q-network architecture must match your environment's state dimensions as input and output one Q-value per possible action, with hidden layers providing the capacity to learn complex state-action value functions.