Quantization Impact on Model Size
What you'll learn: How switching from 32-bit floating point (FP32) to 8-bit integer (INT8) quantization reduces your model's storage footprint by roughly 4×.
The Memory Math
When you quantize a model from FP32 to INT8, you're shrinking each weight and activation from 32 bits down to 8 bits. The math is straightforward:
32 bits ÷ 8 bits = 4× compression
Every parameter that once required 4 bytes now fits in just 1 byte. For a model with millions or billions of parameters, this adds up dramatically.
Concrete Example
Imagine a small convolutional layer with 1 million parameters:
- FP32 storage: 1,000,000 parameters × 4 bytes = 4 MB
- INT8 storage: 1,000,000 parameters × 1 byte = 1 MB
- Savings: 3 MB (75% reduction)
Now scale this to a modern language model like BERT-Base with ~110 million parameters:
- FP32: 110M × 4 bytes ≈ 440 MB
- INT8: 110M × 1 byte ≈ 110 MB
- Savings: ~330 MB
For larger models (billions of parameters), this 4× reduction means the difference between fitting on device memory or not—critical for edge deployment on phones or IoT devices.
The Trade-off
The compression is near-perfect (exactly 4×), but remember: you're trading memory for a small accuracy cost. The discrete 256 values in INT8 can't perfectly represent the continuous range of FP32, introducing quantization error.
Key Takeaway: Quantizing from FP32 to INT8 delivers a consistent 4× model size reduction by using one-quarter the bits per parameter—transforming gigabyte-scale models into hundred-megabyte packages suitable for resource-constrained environments.