Designing the Q-Network in PyTorch
You'll learn how to build a neural network that maps environment states to Q-values for all possible actions.
The Core Idea
In Deep Q-Networks (DQN), we replace the Q-table with a neural network that approximates Q-values. Instead of storing values for every state-action pair (impossible in large or continuous spaces), the Q-network takes a state as input and outputs Q-values for every action simultaneously. Think of it as a function that compresses infinite possibilities into a manageable prediction model.
Network Architecture Fundamentals
Your Q-network needs:
- Input layer: Dimension matches your state representation (e.g., 4 values for CartPole: position, velocity, angle, angular velocity)
- Hidden layers: Typically 1-3 fully connected layers with ReLU activation to learn non-linear patterns
- Output layer: One neuron per action (e.g., 2 outputs for "left" and "right" in CartPole)
PyTorch Implementation
import torch
import torch.nn as nn
class QNetwork(nn.Module):
def __init__(self, state_dim, action_dim, hidden_dim=128):
super(QNetwork, self).__init__()
self.network = nn.Sequential(
nn.Linear(state_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, action_dim)
)
def forward(self, state):
return self.network(state)
This simple architecture takes state vectors, processes them through two hidden layers (128 neurons each), and outputs Q-values for each action—no softmax needed since we're predicting values, not probabilities.
Key Takeaway: The Q-network architecture must match your environment's state dimensions as input and output one Q-value per possible action, with hidden layers providing the capacity to learn complex state-action value functions.