Course contentsShow
← Machine Learning and Deep Learning
Lesson 2953 of 3,53864. GPU Inference OptimizationFree lesson

FP16 and INT8 in Model Formats

Convert and store models in reduced precision formats, understanding format-specific quantization support.

FP16 and INT8 in Model Formats

What You'll Learn

You'll learn how to convert and store models in reduced precision formats (FP16/INT8) and understand how different serialization formats support quantization.

Understanding Precision Reduction in Model Formats

When you export a model to formats like ONNX, TorchScript, or TensorFlow SavedModel, you can choose to store weights in lower precision than the standard FP32 (32-bit floating point). This is like compressing a high-resolution photo—you trade some detail for significantly smaller file size.

FP16 (half-precision) uses 16 bits instead of 32, cutting model size in half. Most modern GPUs have specialized hardware for FP16, making inference faster with minimal accuracy loss for many models.

INT8 (8-bit integer) uses just 8 bits, reducing model size to 25% of the original. This requires quantization—mapping floating-point ranges to integer values. The format must store additional metadata (scale and zero-point values) to convert between representations.

Format-Specific Support

Not all formats handle quantization equally:

  • ONNX supports both FP16 conversion (simple weight casting) and INT8 quantization (requires calibration data and quantization parameters stored in the graph)
  • TorchScript can store FP16 weights directly but requires PyTorch's quantization API for INT8
  • TensorFlow SavedModel has built-in quantization support with TFLite for INT8

The key distinction: FP16 is typically a simple conversion, while INT8 requires calibration to determine optimal scale factors for each layer during the format conversion process.

When converting, you specify the target precision, and the export tool handles weight transformation and inserts any necessary conversion operators in the computation graph.

Key Takeaway: Model formats like ONNX and TorchScript support storing weights in reduced precision (FP16/INT8), with FP16 being a straightforward conversion and INT8 requiring quantization-aware export that embeds calibration metadata directly in the serialized format.