Course contentsShow
← Machine Learning and Deep Learning
Lesson 2270 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

Implementing REINFORCE in PyTorch

Code structure for policy networks, episode collection, loss computation, and parameter updates.

Implementing REINFORCE in PyTorch

What you'll learn: How to structure a complete REINFORCE implementation in PyTorch, from policy network to training loop.

The Complete Picture

Now that you understand REINFORCE conceptually, let's see how all the pieces fit together in code. A PyTorch implementation requires four main components: a policy network, episode collection, loss computation, and parameter updates.

Code Structure

1. Policy Network

Define a neural network that outputs action probabilities. For discrete actions, this typically ends with a softmax layer:

import torch.nn as nn

class PolicyNetwork(nn.Module):
    def __init__(self, state_dim, action_dim):
        super().__init__()
        self.fc = nn.Sequential(
            nn.Linear(state_dim, 128),
            nn.ReLU(),
            nn.Linear(128, action_dim),
            nn.Softmax(dim=-1)
        )
    
    def forward(self, state):
        return self.fc(state)

2. Episode Collection

Roll out a full episode, storing states, actions, and rewards:

def collect_episode(env, policy):
    states, actions, rewards = [], [], []
    state = env.reset()
    done = False
    
    while not done:
        probs = policy(torch.FloatTensor(state))
        action = torch.multinomial(probs, 1).item()
        next_state, reward, done, _ = env.step(action)
        
        states.append(state)
        actions.append(action)
        rewards.append(reward)
        state = next_state
    
    return states, actions, rewards

3. Loss Computation

Calculate returns, then compute the policy gradient loss (negative log probability weighted by return):

def compute_loss(policy, states, actions, returns):
    log_probs = []
    for state, action in zip(states, actions):
        probs = policy(torch.FloatTensor(state))
        log_prob = torch.log(probs[action])
        log_probs.append(log_prob)
    
    loss = -torch.stack([lp * G for lp, G in zip(log_probs, returns)]).sum()
    return loss

4. Training Loop

Collect episodes, compute loss, and update parameters using backpropagation.

Key Takeaway: REINFORCE in PyTorch combines a softmax policy network, episode rollouts, return-weighted log-probability loss, and standard gradient descent—bringing together all prior REINFORCE concepts into executable code.