Quantization Impact on Hardware Needs
What you'll learn: How choosing 8-bit, 4-bit, or lower quantization directly reduces memory and compute requirements, but with trade-offs in output quality.
The Memory-Quality Trade-off
Quantization doesn't just change how your model stores numbers—it fundamentally changes how much hardware you need to run it. When you move from FP16 (16-bit floating point) to INT8 (8-bit integer) or INT4 (4-bit integer), you're not just changing precision—you're cutting memory consumption in half or more.
Real-World Hardware Impact
A 7B parameter model in FP16 requires roughly 14GB of VRAM (7 billion × 2 bytes). Apply 8-bit quantization, and you drop to ~7GB. With 4-bit quantization, you're down to ~3.5GB—now runnable on consumer GPUs like an RTX 3060 (12GB) or even older hardware.
This changes everything: a model that demanded datacenter GPUs can now run on your laptop.
The Quality Cost
Lower precision means less nuance in how weights represent patterns. 8-bit quantization typically shows minimal quality degradation (often <1% on benchmarks). 4-bit quantization introduces more noticeable impacts—slightly less coherent reasoning, occasional vocabulary limitations, or subtle accuracy drops on complex tasks. 2-bit or 3-bit quantization pushes even further, sometimes usable for simpler tasks but with visible quality loss.
Think of it like photo compression: 8-bit is like high-quality JPEG (barely noticeable), 4-bit is moderate compression (good enough for most uses), and 2-bit is heavy compression (you see the artifacts).
Choosing Your Balance
If you have limited VRAM but need reasonable quality, 4-bit quantization (like GPTQ or AWQ) offers the sweet spot. If quality is paramount and hardware allows, stick with 8-bit or FP16.
Key Takeaway: Quantization is your primary lever for fitting larger models onto smaller hardware—8-bit halves memory with minimal quality loss, while 4-bit quarters it with acceptable trade-offs for most applications.