Mixed Precision Training Basics
What you'll learn: How to use torch.cuda.amp to automatically train models faster and with less memory by mixing float16 and float32 computations.
What is Mixed Precision Training?
Mixed precision training means using two different numerical precision levels during training: lower precision (float16) for most operations and higher precision (float32) for numerical stability where needed.
Think of it like using rough sketches (float16) for most of your work—fast and cheap—but switching to detailed drawings (float32) when precision really matters. This gives you speed without sacrificing accuracy.
Why It Matters
Float16 advantages:
- 2x faster computation on modern GPUs
- 2x less memory, allowing larger batch sizes
- Nearly identical final model quality
The catch: Some operations (like loss computation) need float32 to avoid numerical underflow or overflow.
Using torch.cuda.amp
PyTorch's Automatic Mixed Precision (amp) handles the precision switching automatically:
from torch.cuda.amp import autocast, GradScaler
scaler = GradScaler()
for inputs, labels in dataloader:
optimizer.zero_grad()
# Forward pass in mixed precision
with autocast():
outputs = model(inputs)
loss = criterion(outputs, labels)
# Backward pass with gradient scaling
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()
The autocast() context automatically runs operations in float16 when safe, and float32 when necessary. The GradScaler prevents gradients from becoming too small (underflowing) in float16.
Key Takeaway: Mixed precision training with torch.cuda.amp gives you significant speed and memory improvements with minimal code changes—just wrap your forward pass in autocast() and use GradScaler for backpropagation.