Course contentsShow
← Machine Learning and Deep Learning
Lesson 839 of 3,53819. PyTorch FundamentalsFree lesson

Mixed Precision Training Basics

Using torch.cuda.amp for automatic mixed precision to speed up training and reduce memory usage.

Mixed Precision Training Basics

What you'll learn: How to use torch.cuda.amp to automatically train models faster and with less memory by mixing float16 and float32 computations.

What is Mixed Precision Training?

Mixed precision training means using two different numerical precision levels during training: lower precision (float16) for most operations and higher precision (float32) for numerical stability where needed.

Think of it like using rough sketches (float16) for most of your work—fast and cheap—but switching to detailed drawings (float32) when precision really matters. This gives you speed without sacrificing accuracy.

Why It Matters

Float16 advantages:

  • 2x faster computation on modern GPUs
  • 2x less memory, allowing larger batch sizes
  • Nearly identical final model quality

The catch: Some operations (like loss computation) need float32 to avoid numerical underflow or overflow.

Using torch.cuda.amp

PyTorch's Automatic Mixed Precision (amp) handles the precision switching automatically:

from torch.cuda.amp import autocast, GradScaler

scaler = GradScaler()

for inputs, labels in dataloader:
    optimizer.zero_grad()
    
    # Forward pass in mixed precision
    with autocast():
        outputs = model(inputs)
        loss = criterion(outputs, labels)
    
    # Backward pass with gradient scaling
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()

The autocast() context automatically runs operations in float16 when safe, and float32 when necessary. The GradScaler prevents gradients from becoming too small (underflowing) in float16.

Key Takeaway: Mixed precision training with torch.cuda.amp gives you significant speed and memory improvements with minimal code changes—just wrap your forward pass in autocast() and use GradScaler for backpropagation.