Course contentsShow
← Machine Learning and Deep Learning
Lesson 2627 of 3,53857. Model Compression: QuantizationFree lesson

Quantization Error and Rounding

Sources of quantization error: truncation vs rounding strategies and their effect on model accuracy.

Quantization Error and Rounding

What You'll Learn

You'll understand how different rounding strategies affect the precision of quantized models and why this matters for accuracy.

The Core Idea

When you convert floating-point values to lower-bit integers during quantization, you cannot represent every original value exactly—some information is lost. This loss is called quantization error, and it's determined by how you handle numbers that fall between integer steps.

Think of it like measuring temperature: if your thermometer only shows whole degrees, a reading of 72.7°F must become either 72° or 73°. How you decide matters.

Two Main Rounding Strategies

Truncation (Chopping)

Simply drop the decimal part: 72.7 → 72, and -72.7 → -72.

Problem: Truncation introduces a systematic bias. All positive values round down, creating a consistent negative error that can accumulate across layers and harm accuracy.

Rounding to Nearest

Round to the closest integer: 72.7 → 73, 72.3 → 72, and 72.5 → 73 (typically).

Advantage: Errors are more balanced—sometimes slightly high, sometimes slightly low—so they tend to cancel out rather than compound. This usually preserves model accuracy better.

Why It Matters for Model Accuracy

Each quantized weight and activation carries a small error. Over millions of operations, truncation's bias accumulates, potentially shifting the model's predictions significantly. Rounding to nearest distributes errors more evenly, keeping the quantized model's behavior closer to the original.

Most quantization frameworks default to rounding because it empirically yields better accuracy with the same bit-width.

Key Takeaway: Rounding to nearest typically outperforms truncation in quantization because it minimizes systematic bias, keeping quantization error balanced and preserving model accuracy across deep networks.