Course contentsShow
← Machine Learning and Deep Learning
Lesson 712 of 3,53817. Optimization for Deep LearningFree lesson

Implementing Adaptive Optimizers in PyTorch

Practice using torch.optim.Adam, AdamW, and RMSprop with proper hyperparameter configuration and monitoring.

Implementing Adaptive Optimizers in PyTorch

You'll learn how to configure and use Adam, AdamW, and RMSprop optimizers in PyTorch with proper hyperparameter settings.

The Core Idea

Now that you understand why adaptive optimizers work (momentum, per-parameter learning rates, weight decay), it's time to put them into practice. PyTorch's torch.optim module provides ready-to-use implementations of Adam, AdamW, and RMSprop. Your job is to instantiate them correctly, configure their hyperparameters, and monitor their behavior during training.

Setting Up Optimizers in PyTorch

Each optimizer requires your model's parameters and a learning rate. Here's the basic pattern:

import torch.optim as optim

# Adam with default hyperparameters
optimizer = optim.Adam(model.parameters(), lr=0.001, betas=(0.9, 0.999), eps=1e-8)

# AdamW with weight decay
optimizer = optim.AdamW(model.parameters(), lr=0.001, weight_decay=0.01)

# RMSprop
optimizer = optim.RMSprop(model.parameters(), lr=0.01, alpha=0.99, eps=1e-8)

Key hyperparameters:

  • lr: Learning rate (most critical)
  • betas (Adam/AdamW): Momentum coefficients (β₁, β₂)
  • alpha (RMSprop): Decay rate for moving average
  • weight_decay: L2 regularization strength (use with AdamW, not Adam)
  • eps: Small constant for numerical stability

Monitoring During Training

Track the gradient norms to ensure optimization is healthy:

total_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=float('inf'))
print(f"Gradient norm: {total_norm:.4f}")

Key Takeaway: PyTorch's torch.optim makes using adaptive optimizers straightforward—choose Adam or AdamW for most tasks, configure the learning rate carefully, and monitor gradient behavior to catch training issues early.