Course contentsShow
← Machine Learning and Deep Learning
Lesson 2962 of 3,53864. GPU Inference OptimizationFree lesson

INT8 Calibration in TensorRT

Using calibration datasets to determine optimal quantization scales for INT8 inference with minimal accuracy loss.

INT8 Calibration in TensorRT

What you'll learn: How TensorRT uses calibration datasets to find optimal INT8 quantization scales that preserve model accuracy while maximizing performance.

The Calibration Challenge

When you quantize a model from FP32 to INT8, you're compressing values into a much narrower range (256 discrete values instead of billions). The key question is: which 256 values should represent your original distribution? Choose poorly, and important activations get clipped or rounded away. Choose well, and you keep nearly all the model's accuracy while gaining 4x memory reduction and dramatic speedup.

How INT8 Calibration Works

Think of calibration like measuring a room before buying furniture. You need representative measurements to make good decisions.

TensorRT's calibration process:

  1. Run inference on a small, representative calibration dataset (typically 500-1000 samples from your training or validation set)
  2. Collect activation histograms at each layer to understand the distribution of values
  3. Compute optimal scales that minimize information loss—typically using entropy minimization (KL divergence) or percentile methods
  4. Generate a calibration cache storing these scales for each tensor

The calibration dataset must represent real-world input distributions. If you calibrate on cat images but deploy on dog images with different intensity ranges, your scales will be suboptimal.

Calibration Methods

TensorRT offers calibration algorithms:

  • Entropy calibration: Minimizes information loss between FP32 and INT8 distributions
  • MinMax: Simple but can be sensitive to outliers
  • Percentile: Clips extreme values to avoid wasting quantization range

Key Takeaway: INT8 calibration in TensorRT automatically determines per-layer quantization scales by analyzing activation distributions on a representative dataset, enabling near-FP32 accuracy at 4x the speed with minimal manual tuning.