Course contentsShow
← Machine Learning and Deep Learning
Lesson 2775 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

Implementing Mixed Precision with PyTorch AMP

Learn to use torch.cuda.amp.autocast and GradScaler for automatic mixed precision training.

Implementing Mixed Precision with PyTorch AMP

You'll learn how to enable automatic mixed precision training in PyTorch using just two simple components: autocast and GradScaler.

What is PyTorch AMP?

PyTorch's Automatic Mixed Precision (AMP) package (torch.cuda.amp) provides a dead-simple API to implement the mixed precision training algorithm you've learned about. Instead of manually managing FP16/FP32 conversions and loss scaling, AMP handles everything automatically.

The Two Key Components

1. autocast Context Manager

This tells PyTorch which operations should run in FP16 vs FP32. Simply wrap your forward pass:

from torch.cuda.amp import autocast

# Inside training loop
with autocast():
    outputs = model(inputs)
    loss = criterion(outputs, targets)

PyTorch automatically chooses the appropriate precision for each operation—matrix multiplications in FP16 for speed, normalization layers in FP32 for stability.

2. GradScaler for Loss Scaling

This handles the dynamic loss scaling mechanism to prevent gradient underflow:

from torch.cuda.amp import GradScaler

scaler = GradScaler()

# Modified backward pass
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()

The GradScaler multiplies the loss before backward, checks for infs/NaNs, unscales gradients before the optimizer step, and dynamically adjusts the scale factor.

Complete Training Loop Pattern

scaler = GradScaler()

for inputs, targets in dataloader:
    optimizer.zero_grad()
    
    with autocast():
        outputs = model(inputs)
        loss = criterion(outputs, targets)
    
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()

Key Takeaway: PyTorch AMP requires just wrapping your forward pass with autocast() and replacing standard backward/optimizer calls with GradScaler methods—you get 2-3× speedups with minimal code changes and automatic precision management.