Course contentsShow
← Machine Learning and Deep Learning
Lesson 2773 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

Dynamic Loss Scaling Mechanisms

Understand automatic loss scale adjustment based on gradient overflow detection and backoff strategies.

Dynamic Loss Scaling Mechanisms

What you'll learn: How training systems automatically adjust the loss scale factor to prevent gradient underflow without manual tuning.

The Core Problem

In mixed precision training, you scale up the loss before backpropagation to prevent tiny gradients from underflowing to zero in FP16. But how much should you scale? Too little and gradients still vanish; too much and gradients overflow to infinity. Static loss scaling requires manual tuning—dynamic scaling automates this.

How Dynamic Loss Scaling Works

The algorithm monitors your gradients during training and adjusts the scale factor automatically:

  1. Start with a high scale factor (e.g., 2^16 or 65,536)
  2. Check for overflow after computing gradients: if any gradient contains inf or NaN, an overflow occurred
  3. On overflow: Skip the optimizer step, reduce the scale factor (typically halve it), and retry
  4. On success: If a certain number of consecutive iterations pass without overflow (e.g., 2000 steps), increase the scale factor (typically double it)

Think of it like adjusting your car's volume: if someone shouts (overflow), turn it down; if you can't hear for a long time (underflow risk), turn it up gradually.

The Backoff Strategy

The backoff mechanism prevents thrashing. After reducing scale due to overflow, the system waits for sustained stability before increasing again. This patience parameter (often called growth_interval) prevents oscillation between too-high and too-low scales.

The beauty: this happens transparently during training with zero manual intervention.

Key Takeaway: Dynamic loss scaling automatically finds the optimal scale factor by detecting gradient overflow, backing off when necessary, and cautiously increasing during stable periods—eliminating manual tuning while maximizing numerical range utilization.