Course contentsShow
← AI Engineering
Lesson 1039 of 1,88625. Model Serving and Inference OptimizationFree lesson

What is Quantization and Why It Matters

Understanding how quantization reduces model precision to lower memory usage and increase inference speed.

What is Quantization and Why It Matters

What you'll learn: How quantization reduces model precision to save memory and speed up inference without rebuilding your model from scratch.

The Core Idea

Quantization is the process of converting a model's high-precision numbers (typically 32-bit floating point) into lower-precision formats (like 8-bit integers or 16-bit floats). Think of it like compressing a high-resolution photo to a smaller file size—you lose some detail, but the image remains recognizable and takes up far less space.

Why Quantization Matters for LLM Serving

Modern LLMs have billions of parameters. Each parameter in a standard model uses 32 bits (4 bytes) of memory. A 7B parameter model requires roughly 28 GB just to load the weights. With quantization to 8-bit integers, that drops to 7 GB—a 4× reduction.

Memory savings translate directly to:

  • Lower hosting costs (smaller GPU memory requirements)
  • Faster inference (less data movement between memory and compute)
  • Higher throughput (fit more requests in your KV cache budget)
  • Wider deployment (run larger models on consumer hardware)

The Trade-off

Reducing precision means calculations become slightly less accurate. The art of quantization is finding formats that preserve model quality while maximizing efficiency. Modern techniques lose less than 1-2% accuracy for many tasks while delivering massive resource savings.

Key Takeaway: Quantization reduces model memory footprint and inference latency by using lower-precision number formats, making LLM serving more cost-effective and accessible with minimal quality loss.