Course contentsShow
← AI Engineering
Lesson 1053 of 1,88626. Self-Hosted LLM DeploymentFree lesson

llama.cpp: Quantization and Performance Tuning

Understanding quantization levels (Q4, Q5, Q8), trading accuracy for speed, and optimizing inference.

llama.cpp: Quantization and Performance Tuning

You'll learn how to select quantization levels in llama.cpp and tune inference performance for optimal speed-accuracy tradeoffs.

Understanding Quantization Levels

llama.cpp offers multiple quantization formats, commonly referred to by names like Q4_0, Q5_K_M, and Q8_0. The number indicates the bit precision: Q4 uses ~4 bits per weight, Q5 uses ~5 bits, and Q8 uses ~8 bits.

Think of quantization like image compression: a high-resolution photo (Q8) preserves detail but takes more storage, while a compressed version (Q4) loads faster but may lose subtle details.

Trading Accuracy for Speed

Q4 quantization (~4-5 GB for a 7B model) offers the fastest inference and lowest memory usage, ideal for consumer hardware. You'll notice slight quality degradation in complex reasoning tasks.

Q5 quantization (~5-6 GB) balances quality and performance. Most users won't detect meaningful differences from the original model in typical use cases.

Q8 quantization (~7-8 GB) preserves nearly all model quality, suitable when you have sufficient RAM and prioritize accuracy over speed.

Optimization in Practice

When running llama.cpp, you control performance through parameters like thread count (-t), batch size (-b), and context length (-c). More threads leverage CPU parallelism but show diminishing returns beyond physical core counts. Smaller batch sizes reduce memory peaks during generation.

Monitor tokens-per-second (t/s) during inference to gauge performance. A 7B Q4 model might achieve 15-25 t/s on a modern laptop CPU, while Q8 drops to 10-15 t/s due to increased memory bandwidth demands.

Key Takeaway: Choose Q4 for speed-constrained environments, Q5 for balanced production use, and Q8 when accuracy is paramount—llama.cpp's quantization options let you tune the speed-quality tradeoff to your specific hardware and requirements.