INT4 and Sub-Byte Quantization
What You'll Learn
You'll learn how to compress large language models even further by using 4-bit integers and specialized packing techniques that store multiple weights in single bytes.
Why Go Below INT8?
We've seen INT8 quantization reduce model size by 4× compared to FP32. But for massive models like LLMs with billions of parameters, we can push further. INT4 quantization represents each weight using only 4 bits (16 possible values), achieving an 8× compression ratio. Sub-byte formats (3-bit, 2-bit) go even smaller.
The Core Challenge
Standard computer architectures operate on byte-aligned data. A 4-bit value is half a byte, creating two problems:
- Memory packing: We must pack two INT4 values into one byte
- Computation: Standard hardware lacks native INT4 arithmetic instructions
Implementation Strategies
Bit-Level Packing
Store two INT4 weights in a single uint8:
- Upper 4 bits hold weight₁
- Lower 4 bits hold weight₂
This halves memory footprint but requires unpacking before computation.
Specialized Kernels
Since CPUs/GPUs don't natively multiply INT4 numbers, we need custom kernels that:
- Unpack INT4 pairs from bytes
- Perform computations (often unpacking to INT8 internally)
- Leverage SIMD instructions for parallel unpacking
Group Quantization
With only 16 quantization levels, per-tensor quantization loses too much precision. Per-group quantization (e.g., 128 consecutive weights share scale/zero-point) balances accuracy and overhead.
Practical Trade-offs
INT4 works surprisingly well for LLM inference when combined with careful calibration and per-group strategies. The accuracy loss is often acceptable (1-3% degradation) for the 2× memory savings over INT8.
Key Takeaway: INT4 quantization achieves extreme compression by packing two 4-bit weights per byte and using specialized kernels with group-wise quantization parameters to maintain acceptable accuracy in large language models.