Implementing Adaptive Optimizers in PyTorch
You'll learn how to configure and use Adam, AdamW, and RMSprop optimizers in PyTorch with proper hyperparameter settings.
The Core Idea
Now that you understand why adaptive optimizers work (momentum, per-parameter learning rates, weight decay), it's time to put them into practice. PyTorch's torch.optim module provides ready-to-use implementations of Adam, AdamW, and RMSprop. Your job is to instantiate them correctly, configure their hyperparameters, and monitor their behavior during training.
Setting Up Optimizers in PyTorch
Each optimizer requires your model's parameters and a learning rate. Here's the basic pattern:
import torch.optim as optim
# Adam with default hyperparameters
optimizer = optim.Adam(model.parameters(), lr=0.001, betas=(0.9, 0.999), eps=1e-8)
# AdamW with weight decay
optimizer = optim.AdamW(model.parameters(), lr=0.001, weight_decay=0.01)
# RMSprop
optimizer = optim.RMSprop(model.parameters(), lr=0.01, alpha=0.99, eps=1e-8)
Key hyperparameters:
lr: Learning rate (most critical)betas(Adam/AdamW): Momentum coefficients (β₁, β₂)alpha(RMSprop): Decay rate for moving averageweight_decay: L2 regularization strength (use with AdamW, not Adam)eps: Small constant for numerical stability
Monitoring During Training
Track the gradient norms to ensure optimization is healthy:
total_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=float('inf'))
print(f"Gradient norm: {total_norm:.4f}")
Key Takeaway: PyTorch's torch.optim makes using adaptive optimizers straightforward—choose Adam or AdamW for most tasks, configure the learning rate carefully, and monitor gradient behavior to catch training issues early.