4-bit NormalFloat (NF4) Quantization
What you'll learn: How NF4 uses just 4 bits per weight by exploiting the normal distribution pattern found in neural network weights.
The Core Idea
Standard quantization methods (like uniform 4-bit) divide the numerical range into equally-spaced buckets. But neural network weights aren't uniformly distributed—they follow a normal (Gaussian) distribution, clustering heavily around zero with fewer values at the extremes.
NF4 is a specialized data type designed specifically for this reality. Instead of wasting precious bits on rarely-used extreme values, NF4 allocates more quantization levels to the dense central region where most weights live.
How NF4 Works
Think of it like parking spaces in a city: you don't put equal parking everywhere—you concentrate spaces where people actually go. NF4 does this by:
- Assuming weights are normally distributed (zero-centered)
- Placing quantization bins closer together near zero (where most weights cluster)
- Spacing bins farther apart at the tails (where few weights exist)
This gives you better precision where it matters most, even with only 16 possible values (2^4 = 16).
Why QLoRA Uses NF4
QLoRA specifically chose NF4 because:
- Pretrained LLM weights empirically follow normal distributions
- You get better accuracy than uniform 4-bit quantization
- It still uses only 4 bits per parameter (75% memory reduction vs FP16)
- The information-theoretically optimal encoding for normally distributed data
Key Takeaway: NF4 is a 4-bit quantization format optimized for normally distributed weights, achieving better accuracy than uniform quantization by concentrating precision where weight values are most dense—making it ideal for QLoRA's memory-efficient fine-tuning.