Course contentsShow
← AI Engineering
Lesson 78 of 1,8862. Working with Pre-trained ModelsFree lesson

What is Model Quantization

Understand quantization fundamentals: reducing precision from FP32 to INT8 to decrease model size and latency.

What is Model Quantization

What you'll learn: How quantization reduces model size and speeds up inference by using lower-precision numbers.

The Core Idea

Model quantization is the process of converting your model's weights and activations from high-precision numbers (like 32-bit floating point, FP32) to lower-precision representations (like 8-bit integers, INT8).

Think of it like switching from using exact decimal measurements (3.14159265...) to rounded whole numbers (3). You lose some precision, but the numbers become much simpler to store and calculate.

Why This Matters

When you load a pre-trained model from Hugging Face, its weights are typically stored as FP32 values. Each weight takes 32 bits (4 bytes) of memory. A model with 1 billion parameters needs roughly 4GB of memory just for weights!

With INT8 quantization:

  • Each weight uses only 8 bits (1 byte)
  • That same 1 billion parameter model now needs just 1GB
  • 4x smaller model size
  • Faster computation because integer math is simpler than floating-point math

The Trade-off

Just like rounding 3.14159 to 3 loses information, quantization reduces numerical precision. Most models handle this surprisingly well—you typically see only minor accuracy drops (1-2%) while gaining significant speed and memory benefits.

This connects directly to your earlier work with memory management and GPU allocation (#77)—quantization is one of the most powerful tools for fitting larger models into limited memory.

Key Takeaway: Quantization converts model weights from 32-bit floats to 8-bit integers, reducing model size by ~75% and speeding up inference with minimal accuracy loss—a critical technique when working with memory constraints.