Course contentsShow
← Machine Learning and Deep Learning
Lesson 732 of 3,53817. Optimization for Deep LearningFree lesson

Mixed Precision and Gradient Scaling

Preventing gradient underflow in FP16 training using loss scaling and dynamic gradient adjustment.

Mixed Precision and Gradient Scaling

What You'll Learn

You'll learn how to prevent tiny gradients from disappearing when training neural networks with lower-precision floating-point numbers (FP16).

The Problem with Half Precision

Modern deep learning uses mixed precision training—computing some operations in FP16 (16-bit floats) instead of FP32 (32-bit floats) to speed up training and reduce memory usage. However, FP16 has a much smaller range than FP32: it can only represent values roughly between 6×10⁻⁸ and 65,000.

During backpropagation, gradients can become very small—sometimes smaller than FP16's minimum representable value. When this happens, they underflow to zero, and your network stops learning. It's like trying to measure teaspoons of water with a bucket: the measurement tool just isn't precise enough.

Loss Scaling to the Rescue

The solution is loss scaling: before backpropagation, multiply your loss by a large number (e.g., 1024 or 2048). This scales up all the gradients proportionally, bringing tiny values into FP16's representable range.

After computing the scaled gradients, you divide them by the same scaling factor before updating weights, restoring their true magnitude. Think of it as temporarily zooming in with a magnifying glass to see tiny details, then zooming back out.

Dynamic Gradient Adjustment

Dynamic loss scaling automatically adjusts the scale factor during training. If gradients overflow (become too large), it reduces the scale. If training is stable, it gradually increases the scale to maximize numerical precision. This happens transparently as you train.

Key Takeaway: Mixed precision training uses loss scaling to multiply gradients before backpropagation and divide them afterward, preventing gradient underflow in FP16 while maintaining FP32 numerical stability where it matters.