Course contentsShow
← Machine Learning and Deep Learning
Lesson 2662 of 3,53857. Model Compression: QuantizationFree lesson

INT4 and Sub-Byte Quantization

Implementing 4-bit and lower quantization with specialized kernels and packing strategies for LLMs.

INT4 and Sub-Byte Quantization

What You'll Learn

You'll learn how to compress large language models even further by using 4-bit integers and specialized packing techniques that store multiple weights in single bytes.

Why Go Below INT8?

We've seen INT8 quantization reduce model size by 4× compared to FP32. But for massive models like LLMs with billions of parameters, we can push further. INT4 quantization represents each weight using only 4 bits (16 possible values), achieving an 8× compression ratio. Sub-byte formats (3-bit, 2-bit) go even smaller.

The Core Challenge

Standard computer architectures operate on byte-aligned data. A 4-bit value is half a byte, creating two problems:

  1. Memory packing: We must pack two INT4 values into one byte
  2. Computation: Standard hardware lacks native INT4 arithmetic instructions

Implementation Strategies

Bit-Level Packing

Store two INT4 weights in a single uint8:

  • Upper 4 bits hold weight₁
  • Lower 4 bits hold weight₂

This halves memory footprint but requires unpacking before computation.

Specialized Kernels

Since CPUs/GPUs don't natively multiply INT4 numbers, we need custom kernels that:

  • Unpack INT4 pairs from bytes
  • Perform computations (often unpacking to INT8 internally)
  • Leverage SIMD instructions for parallel unpacking

Group Quantization

With only 16 quantization levels, per-tensor quantization loses too much precision. Per-group quantization (e.g., 128 consecutive weights share scale/zero-point) balances accuracy and overhead.

Practical Trade-offs

INT4 works surprisingly well for LLM inference when combined with careful calibration and per-group strategies. The accuracy loss is often acceptable (1-3% degradation) for the 2× memory savings over INT8.

Key Takeaway: INT4 quantization achieves extreme compression by packing two 4-bit weights per byte and using specialized kernels with group-wise quantization parameters to maintain acceptable accuracy in large language models.