Implementing Mixed Precision with PyTorch AMP
You'll learn how to enable automatic mixed precision training in PyTorch using just two simple components: autocast and GradScaler.
What is PyTorch AMP?
PyTorch's Automatic Mixed Precision (AMP) package (torch.cuda.amp) provides a dead-simple API to implement the mixed precision training algorithm you've learned about. Instead of manually managing FP16/FP32 conversions and loss scaling, AMP handles everything automatically.
The Two Key Components
1. autocast Context Manager
This tells PyTorch which operations should run in FP16 vs FP32. Simply wrap your forward pass:
from torch.cuda.amp import autocast
# Inside training loop
with autocast():
outputs = model(inputs)
loss = criterion(outputs, targets)
PyTorch automatically chooses the appropriate precision for each operation—matrix multiplications in FP16 for speed, normalization layers in FP32 for stability.
2. GradScaler for Loss Scaling
This handles the dynamic loss scaling mechanism to prevent gradient underflow:
from torch.cuda.amp import GradScaler
scaler = GradScaler()
# Modified backward pass
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()
The GradScaler multiplies the loss before backward, checks for infs/NaNs, unscales gradients before the optimizer step, and dynamically adjusts the scale factor.
Complete Training Loop Pattern
scaler = GradScaler()
for inputs, targets in dataloader:
optimizer.zero_grad()
with autocast():
outputs = model(inputs)
loss = criterion(outputs, targets)
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()
Key Takeaway: PyTorch AMP requires just wrapping your forward pass with autocast() and replacing standard backward/optimizer calls with GradScaler methods—you get 2-3× speedups with minimal code changes and automatic precision management.