Quantization Error and Rounding
What You'll Learn
You'll understand how different rounding strategies affect the precision of quantized models and why this matters for accuracy.
The Core Idea
When you convert floating-point values to lower-bit integers during quantization, you cannot represent every original value exactly—some information is lost. This loss is called quantization error, and it's determined by how you handle numbers that fall between integer steps.
Think of it like measuring temperature: if your thermometer only shows whole degrees, a reading of 72.7°F must become either 72° or 73°. How you decide matters.
Two Main Rounding Strategies
Truncation (Chopping)
Simply drop the decimal part: 72.7 → 72, and -72.7 → -72.
Problem: Truncation introduces a systematic bias. All positive values round down, creating a consistent negative error that can accumulate across layers and harm accuracy.
Rounding to Nearest
Round to the closest integer: 72.7 → 73, 72.3 → 72, and 72.5 → 73 (typically).
Advantage: Errors are more balanced—sometimes slightly high, sometimes slightly low—so they tend to cancel out rather than compound. This usually preserves model accuracy better.
Why It Matters for Model Accuracy
Each quantized weight and activation carries a small error. Over millions of operations, truncation's bias accumulates, potentially shifting the model's predictions significantly. Rounding to nearest distributes errors more evenly, keeping the quantized model's behavior closer to the original.
Most quantization frameworks default to rounding because it empirically yields better accuracy with the same bit-width.
Key Takeaway: Rounding to nearest typically outperforms truncation in quantization because it minimizes systematic bias, keeping quantization error balanced and preserving model accuracy across deep networks.