Course contentsShow
← Machine Learning and Deep Learning
Lesson 2325 of 3,53850. Deep Reinforcement Learning: Advanced Policy MethodsFree lesson

Implementing Continuous Control in PyTorch

Building actor and critic networks, computing policy gradients, and handling continuous action clipping.

Implementing Continuous Control in PyTorch

What you'll learn: How to build actor-critic networks for continuous control, compute policy gradients, and safely clip continuous actions in PyTorch.

The Core Implementation

Now that you understand DDPG, TD3, and SAC conceptually, it's time to implement the key components. Continuous control requires careful handling of unbounded action outputs and proper network architectures.

Building Actor and Critic Networks

The actor network outputs continuous action values (often using tanh activation to bound outputs to [-1, 1]). The critic network takes both states and actions as input to estimate Q-values:

class Actor(nn.Module):
    def __init__(self, state_dim, action_dim, max_action):
        super(Actor, self).__init__()
        self.layer1 = nn.Linear(state_dim, 256)
        self.layer2 = nn.Linear(256, 256)
        self.layer3 = nn.Linear(256, action_dim)
        self.max_action = max_action
    
    def forward(self, state):
        x = F.relu(self.layer1(state))
        x = F.relu(self.layer2(x))
        return self.max_action * torch.tanh(self.layer3(x))

class Critic(nn.Module):
    def __init__(self, state_dim, action_dim):
        super(Critic, self).__init__()
        self.layer1 = nn.Linear(state_dim + action_dim, 256)
        self.layer2 = nn.Linear(256, 256)
        self.layer3 = nn.Linear(256, 1)
    
    def forward(self, state, action):
        x = torch.cat([state, action], dim=1)
        x = F.relu(self.layer1(x))
        x = F.relu(self.layer2(x))
        return self.layer3(x)

Computing Policy Gradients

The deterministic policy gradient updates the actor to maximize Q-values:

actor_loss = -critic(state, actor(state)).mean()
actor_optimizer.zero_grad()
actor_loss.backward()
actor_optimizer.step()

Action Clipping

Always clip actions to respect environment bounds:

action = actor(state).cpu().data.numpy()
action = np.clip(action, env.action_space.low, env.action_space.high)

Key Takeaway: Implementing continuous control requires actor networks with bounded outputs, critics that accept state-action pairs, computing gradients that maximize Q-values, and explicit action clipping to respect environment constraints.