Quantization Basics for Fine-Tuning
What You'll Learn
You'll learn how quantization dramatically reduces memory usage during fine-tuning by storing model weights in lower precision formats like 4-bit or 8-bit instead of full 32-bit floating point.
The Memory Problem in Fine-Tuning
When fine-tuning large language models, memory is your primary constraint. A 7B parameter model stored in standard 32-bit floating point requires roughly 28GB of memory just for the weights. Add gradients, optimizer states, and activations, and you quickly exceed consumer GPU capacity.
What is Quantization?
Quantization means representing numbers with fewer bits. Instead of storing each weight as a 32-bit float (FP32), you can use:
- 8-bit integers (INT8): 4× memory reduction
- 4-bit integers (INT4): 8× memory reduction
Analogy: Imagine measuring room temperature. You could use precise decimal readings like 72.3847°F (high precision), or round to whole degrees like 72°F (lower precision but still useful). Quantization does this for neural network weights.
How It Works for Fine-Tuning
During fine-tuning with quantization:
- Base weights are stored in low precision (4-bit or 8-bit)
- Computation happens at higher precision when needed
- Trainable parameters (like LoRA adapters) remain at full precision
This hybrid approach lets you fine-tune massive models on limited hardware. A quantized 7B model might fit in 4-7GB instead of 28GB, making single-GPU fine-tuning feasible.
Trade-offs
Lower precision means slight accuracy loss in weight representation, but research shows 8-bit and even 4-bit quantization preserve model quality remarkably well for most tasks.
Key Takeaway: Quantization reduces fine-tuning memory footprint by 4-8× by storing weights in lower-bit formats (4-bit/8-bit), enabling parameter-efficient methods like LoRA to run on consumer hardware without significant quality loss.