Course contentsShow
← Machine Learning and Deep Learning
Lesson 2772 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

Loss Scaling: Preventing Gradient Underflow

Learn how dynamic and static loss scaling prevents gradients from becoming zero in FP16.

Loss Scaling: Preventing Gradient Underflow

You'll learn how loss scaling prevents tiny gradients from vanishing to zero when training with FP16 precision.

The Problem: Gradient Underflow

When using FP16 (16-bit floating point) in mixed precision training, you gain speed and memory efficiency—but there's a catch. FP16 can only represent numbers down to about 6×10⁻⁸. Many gradients during backpropagation are smaller than this threshold, so they underflow to zero. When gradients become zero, your model stops learning.

How Loss Scaling Works

Loss scaling is elegantly simple: multiply your loss by a large scalar (like 1000 or 65536) before backpropagation. This shifts all gradients into FP16's representable range.

The workflow:

  1. Compute loss (FP32)
  2. Multiply loss by scale factor (e.g., 1024)
  3. Backward pass computes scaled gradients (FP16)
  4. Unscale gradients back (divide by 1024) before optimizer step
  5. Update weights (FP32)

Think of it like using a magnifying glass: you magnify tiny values so they're visible, work with them, then scale back to their true size.

Static vs Dynamic Loss Scaling

Static scaling uses a fixed multiplier throughout training. Simple but risky—if gradients are too large after scaling, you get overflow (infinity).

Dynamic scaling automatically adjusts the scale factor. Start with a large scale (e.g., 65536). If overflow is detected (inf/NaN in gradients), reduce the scale by half and skip that update. If training is stable for many iterations, gradually increase the scale. This adaptive approach works robustly across different models and phases of training.

Key Takeaway: Loss scaling multiplies the loss before backpropagation to keep FP16 gradients above the underflow threshold, then unscales before the optimizer step—dynamic scaling adapts automatically to prevent both underflow and overflow.