Why Mixed Precision Training Works
What you'll learn: How neural networks maintain accuracy when using lower-precision arithmetic through strategic gradient scaling techniques.
The Precision Paradox
At first glance, it seems paradoxical: you reduce numerical precision from FP32 (32-bit floating point) to FP16 (16-bit), yet your model still converges to nearly the same accuracy. How?
The key insight is that not all numbers in training need the same precision. Neural networks are remarkably robust to noise during training—small perturbations in weights and activations don't derail learning. FP16 provides enough precision for forward passes and most backward computations.
The Gradient Challenge
The critical problem is gradients can be extremely small. FP16's limited range (approximately 6×10⁻⁸ to 65,504) causes tiny gradients to underflow to zero—a phenomenon called "gradient vanishing due to precision." When gradients disappear, learning stops.
Gradient Scaling: The Solution
Mixed precision solves this through loss scaling:
- Scale up the loss before backpropagation (multiply by a large factor, e.g., 1024 or 65536)
- Gradients are automatically scaled through the chain rule
- Scaled gradients stay within FP16's representable range
- Unscale gradients before the optimizer step
- Update weights in FP32 (the "master copy")
Think of it like photography in low light: you amplify the signal (increase ISO) to capture detail, then adjust exposure in post-processing. Gradient scaling amplifies tiny values so they survive FP16 computation.
Why Accuracy Is Preserved
Forward passes and weight updates happen in FP32 or use FP16 with sufficient precision. The accumulation of tiny errors in FP16 is managed by maintaining master weights in FP32 and only using FP16 for computation-heavy operations (matrix multiplies, convolutions). Statistical noise from reduced precision acts like a mild regularizer—often harmless or even beneficial.
Key Takeaway: Mixed precision training works by strategically scaling gradients to prevent underflow in FP16 arithmetic while maintaining a high-precision master copy of weights, allowing 2-3× speedups with minimal accuracy loss.