Implementing Continuous Control in PyTorch
What you'll learn: How to build actor-critic networks for continuous control, compute policy gradients, and safely clip continuous actions in PyTorch.
The Core Implementation
Now that you understand DDPG, TD3, and SAC conceptually, it's time to implement the key components. Continuous control requires careful handling of unbounded action outputs and proper network architectures.
Building Actor and Critic Networks
The actor network outputs continuous action values (often using tanh activation to bound outputs to [-1, 1]). The critic network takes both states and actions as input to estimate Q-values:
class Actor(nn.Module):
def __init__(self, state_dim, action_dim, max_action):
super(Actor, self).__init__()
self.layer1 = nn.Linear(state_dim, 256)
self.layer2 = nn.Linear(256, 256)
self.layer3 = nn.Linear(256, action_dim)
self.max_action = max_action
def forward(self, state):
x = F.relu(self.layer1(state))
x = F.relu(self.layer2(x))
return self.max_action * torch.tanh(self.layer3(x))
class Critic(nn.Module):
def __init__(self, state_dim, action_dim):
super(Critic, self).__init__()
self.layer1 = nn.Linear(state_dim + action_dim, 256)
self.layer2 = nn.Linear(256, 256)
self.layer3 = nn.Linear(256, 1)
def forward(self, state, action):
x = torch.cat([state, action], dim=1)
x = F.relu(self.layer1(x))
x = F.relu(self.layer2(x))
return self.layer3(x)
Computing Policy Gradients
The deterministic policy gradient updates the actor to maximize Q-values:
actor_loss = -critic(state, actor(state)).mean()
actor_optimizer.zero_grad()
actor_loss.backward()
actor_optimizer.step()
Action Clipping
Always clip actions to respect environment bounds:
action = actor(state).cpu().data.numpy()
action = np.clip(action, env.action_space.low, env.action_space.high)
Key Takeaway: Implementing continuous control requires actor networks with bounded outputs, critics that accept state-action pairs, computing gradients that maximize Q-values, and explicit action clipping to respect environment constraints.