Course contentsShow
← AI Engineering
Lesson 1067 of 1,88626. Self-Hosted LLM DeploymentFree lesson

Quantization Impact on Hardware Needs

Understand how 8-bit, 4-bit, and lower quantization reduce memory requirements while affecting quality.

Quantization Impact on Hardware Needs

What you'll learn: How choosing 8-bit, 4-bit, or lower quantization directly reduces memory and compute requirements, but with trade-offs in output quality.

The Memory-Quality Trade-off

Quantization doesn't just change how your model stores numbers—it fundamentally changes how much hardware you need to run it. When you move from FP16 (16-bit floating point) to INT8 (8-bit integer) or INT4 (4-bit integer), you're not just changing precision—you're cutting memory consumption in half or more.

Real-World Hardware Impact

A 7B parameter model in FP16 requires roughly 14GB of VRAM (7 billion × 2 bytes). Apply 8-bit quantization, and you drop to ~7GB. With 4-bit quantization, you're down to ~3.5GB—now runnable on consumer GPUs like an RTX 3060 (12GB) or even older hardware.

This changes everything: a model that demanded datacenter GPUs can now run on your laptop.

The Quality Cost

Lower precision means less nuance in how weights represent patterns. 8-bit quantization typically shows minimal quality degradation (often <1% on benchmarks). 4-bit quantization introduces more noticeable impacts—slightly less coherent reasoning, occasional vocabulary limitations, or subtle accuracy drops on complex tasks. 2-bit or 3-bit quantization pushes even further, sometimes usable for simpler tasks but with visible quality loss.

Think of it like photo compression: 8-bit is like high-quality JPEG (barely noticeable), 4-bit is moderate compression (good enough for most uses), and 2-bit is heavy compression (you see the artifacts).

Choosing Your Balance

If you have limited VRAM but need reasonable quality, 4-bit quantization (like GPTQ or AWQ) offers the sweet spot. If quality is paramount and hardware allows, stick with 8-bit or FP16.

Key Takeaway: Quantization is your primary lever for fitting larger models onto smaller hardware—8-bit halves memory with minimal quality loss, while 4-bit quarters it with acceptable trade-offs for most applications.