Course contentsShow
← AI Engineering
Lesson 1354 of 1,88633. Parameter-Efficient Fine-Tuning (PEFT)Free lesson

NF4 Quantization and Double Quantization

Understanding the Normal Float 4-bit format and nested quantization techniques used in QLoRA.

NF4 Quantization and Double Quantization

What You'll Learn

You'll understand how QLoRA achieves extreme memory efficiency through Normal Float 4-bit quantization and a clever nested quantization trick.

The Core Idea

In QLoRA, the base model isn't stored in standard 16-bit floats—it's compressed to just 4 bits using a special format called NF4 (Normal Float 4). This format is optimized for weights that follow a normal distribution, which most neural network parameters do.

Think of NF4 like a smart compression algorithm that knows neural networks tend to have lots of weights clustered around zero. Instead of evenly spacing the 16 possible values (0-15) across a range, NF4 places more "bins" near zero where most weights live, and fewer bins at the extremes. This preserves the important information while using minimal bits.

Double Quantization: Quantizing the Quantizer

But there's a twist! When you quantize weights, you need to store quantization constants (scaling factors) for each block of weights. These constants themselves take up memory.

Double quantization solves this by quantizing those quantization constants too. It's like compression within compression—you quantize the main weights using NF4, then quantize the scaling factors from 32-bit down to 8-bit. This saves an additional ~0.37 bits per parameter on average.

The Memory Win

Without these techniques, a 7B parameter model needs ~14GB (2 bytes per param). With NF4 + double quantization, it drops to ~3.5GB—a 4x reduction—while maintaining nearly the same training quality when combined with LoRA adapters.

Key Takeaway: NF4 quantization exploits the normal distribution of neural network weights for efficient 4-bit storage, while double quantization applies a second layer of compression to the quantization metadata itself, enabling QLoRA to fine-tune massive models on consumer GPUs.