Course contentsShow
← Machine Learning and Deep Learning
Lesson 2771 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

The Mixed Precision Training Algorithm

Master the core workflow: FP16 forward pass, loss scaling, FP32 master weights, and gradient updates.

The Mixed Precision Training Algorithm

What you'll learn: How mixed precision training orchestrates FP16 computation with FP32 stability to accelerate deep learning without losing accuracy.

The Core Workflow

Mixed precision training isn't just "use FP16 everywhere." It's a carefully choreographed algorithm that keeps computation fast while maintaining numerical stability through strategic precision management.

Here's the complete loop:

1. Maintain FP32 Master Weights

Your model parameters live in full FP32 precision as the "source of truth." These master weights preserve small gradient updates that would vanish in FP16.

2. FP16 Forward Pass

Before each forward pass, cast a copy of weights to FP16. All activations and intermediate computations happen in FP16, leveraging Tensor Cores for 2-8× speedup.

3. Loss Scaling

Multiply the loss by a large scalar (e.g., 1024 or 65536) before backpropagation. This shifts tiny gradients into FP16's representable range, preventing underflow. Think of it as amplifying a whisper so it's audible.

4. FP16 Backward Pass

Compute scaled gradients in FP16. The loss scaling ensures gradients that would normally underflow (become zero) remain representable.

5. Unscale and Check

Convert gradients to FP32, then divide by the loss scale to get true gradient values. Check for infinities or NaNs—if found, skip this update and reduce the loss scale.

6. FP32 Optimizer Update

Apply gradients to FP32 master weights using your optimizer. Even tiny gradient contributions accumulate properly in full precision.

Key Takeaway: Mixed precision training uses FP16 for speed during forward/backward passes while maintaining FP32 master weights and strategically scaling losses to prevent gradient underflow—giving you the best of both precision worlds.