Dynamic Range and Clipping
What You'll Learn
You'll learn how choosing quantization range boundaries (min/max values) affects accuracy, and why outliers can sabotage your quantized model.
The Core Problem
When quantizing weights or activations, you must map floating-point values into a limited integer range (say, 0-255 for 8-bit). But what min and max values should define your range?
Too narrow: You clip (truncate) extreme values, losing information.
Too wide: You waste precious quantization levels on rarely-used ranges, losing precision where it matters.
Think of it like adjusting exposure on a camera. If you set the range for bright sunlight when photographing a dimly-lit room, everything looks dark and loses detail. If you optimize for shadows, the bright spots "blow out" (clip).
Calibration Methods
Min-Max Calibration: Use the actual minimum and maximum values observed in your data. Simple but vulnerable—a single outlier stretches your entire range.
Percentile Clipping: Ignore the extreme 0.01% or 1% of values. You intentionally clip outliers to preserve precision for the majority of data. This trades a tiny bit of clipping error for much better resolution in the typical range.
KL Divergence / Entropy Calibration: Find the range that minimizes information loss between original and quantized distributions—more sophisticated but computationally expensive.
Why Outliers Matter
A single weight value of 100.0 among thousands of values around ±1.0 forces you to spread your 256 quantization levels across [-100, 100] instead of [-1, 1]. Most values then map to nearly identical integers, obliterating useful distinctions.
Key Takeaway: Choosing quantization min/max values is a calibration problem—you must balance clipping outliers against preserving precision in the typical range, often using percentile-based methods to prevent extreme values from degrading overall accuracy.