Course contentsShow
← Machine Learning and Deep Learning
Lesson 2638 of 3,53857. Model Compression: QuantizationFree lesson

Entropy-Based Calibration (KL Divergence)

Learn advanced calibration that minimizes information loss by finding quantization parameters that preserve activation distributions.

Entropy-Based Calibration (KL Divergence)

What you'll learn: How to find optimal quantization parameters by minimizing the difference between original and quantized activation distributions using KL divergence.

The Core Idea

While MinMax and Percentile calibration methods focus on range-based heuristics, entropy-based calibration takes a distribution-aware approach. It searches for the best quantization parameters (scale and zero-point) by measuring how much information is lost when you quantize your activations.

The key metric is KL divergence (Kullback-Leibler divergence), which measures the difference between two probability distributions. Think of it like comparing two histograms: if the original floating-point activations follow one distribution and the quantized activations follow another, KL divergence tells you how "far apart" these distributions are.

How It Works

  1. Collect activation statistics during calibration passes (like other methods)
  2. Build a histogram of the original activation distribution
  3. Try different quantization ranges (different clipping thresholds)
  4. For each candidate range:
    • Simulate quantization with those parameters
    • Create a histogram of the quantized distribution
    • Calculate KL divergence between original and quantized
  5. Choose the range with minimum KL divergence — this preserves the most information

Why This Matters

Unlike simple MinMax (which often overshoots) or Percentile (which uses fixed cutoffs), KL divergence adapts to each layer's actual activation pattern. For layers with long-tailed distributions or outliers, this can prevent unnecessary precision loss by finding a sweet spot between clipping outliers and preserving typical values.

Key Takeaway: Entropy-based calibration using KL divergence finds quantization parameters that best preserve the statistical distribution of activations, minimizing information loss compared to simpler range-based methods.