Implementing REINFORCE in PyTorch
What you'll learn: How to structure a complete REINFORCE implementation in PyTorch, from policy network to training loop.
The Complete Picture
Now that you understand REINFORCE conceptually, let's see how all the pieces fit together in code. A PyTorch implementation requires four main components: a policy network, episode collection, loss computation, and parameter updates.
Code Structure
1. Policy Network
Define a neural network that outputs action probabilities. For discrete actions, this typically ends with a softmax layer:
import torch.nn as nn
class PolicyNetwork(nn.Module):
def __init__(self, state_dim, action_dim):
super().__init__()
self.fc = nn.Sequential(
nn.Linear(state_dim, 128),
nn.ReLU(),
nn.Linear(128, action_dim),
nn.Softmax(dim=-1)
)
def forward(self, state):
return self.fc(state)
2. Episode Collection
Roll out a full episode, storing states, actions, and rewards:
def collect_episode(env, policy):
states, actions, rewards = [], [], []
state = env.reset()
done = False
while not done:
probs = policy(torch.FloatTensor(state))
action = torch.multinomial(probs, 1).item()
next_state, reward, done, _ = env.step(action)
states.append(state)
actions.append(action)
rewards.append(reward)
state = next_state
return states, actions, rewards
3. Loss Computation
Calculate returns, then compute the policy gradient loss (negative log probability weighted by return):
def compute_loss(policy, states, actions, returns):
log_probs = []
for state, action in zip(states, actions):
probs = policy(torch.FloatTensor(state))
log_prob = torch.log(probs[action])
log_probs.append(log_prob)
loss = -torch.stack([lp * G for lp, G in zip(log_probs, returns)]).sum()
return loss
4. Training Loop
Collect episodes, compute loss, and update parameters using backpropagation.
Key Takeaway: REINFORCE in PyTorch combines a softmax policy network, episode rollouts, return-weighted log-probability loss, and standard gradient descent—bringing together all prior REINFORCE concepts into executable code.