What is Quantization and Why It Matters
You'll learn: How quantization reduces model size and speeds up inference by mapping high-precision numbers to discrete, lower-precision values.
The Core Idea
Imagine you're measuring water temperature. A digital thermometer might show 72.4839261°F with ridiculous precision, but for most purposes, saying "about 72°F" works perfectly fine. Quantization does the same thing for neural network weights and activations—it trades ultra-precise numbers for simpler approximations.
In deep learning, models typically store weights as 32-bit floating-point numbers (FP32), which can represent values with high precision but consume lots of memory. Quantization maps these continuous values to a smaller set of discrete levels, often 8-bit integers (INT8) or even fewer bits.
Why It Matters
Memory Footprint
A quantized INT8 model uses 4× less memory than its FP32 counterpart. This makes deployment feasible on mobile devices, edge hardware, and embedded systems where resources are limited.
Inference Speed
Integer operations are faster than floating-point arithmetic on most hardware, especially specialized accelerators. Quantized models can run 2-4× faster while maintaining acceptable accuracy.
Energy Efficiency
Smaller models with simpler operations consume less power—critical for battery-powered devices and data centers optimizing for sustainability.
The Trade-off
You lose some precision. A model trained with high-precision floats might perform slightly worse when quantized, but modern techniques (which you'll learn) minimize this accuracy drop, often keeping it under 1%.
Key Takeaway: Quantization compresses models by mapping continuous floating-point values to discrete lower-precision representations, dramatically reducing size and increasing speed with minimal accuracy loss—essential for deploying models in resource-constrained environments.