GPTQ: Weight-Only Quantization for LLMs
You'll learn how GPTQ enables aggressive weight compression to 4-bit or even lower, dramatically reducing LLM memory footprint while maintaining quality.
What is GPTQ?
GPTQ (Generalized Post-Training Quantization) is a specialized post-training quantization method designed specifically for large language models. Unlike basic PTQ approaches that might quantize everything uniformly, GPTQ focuses exclusively on compressing model weights while keeping activations at higher precision.
Think of it like compressing a high-resolution photo: GPTQ identifies which "pixels" (weights) can be safely reduced in quality without ruining the overall image (model performance). It does this by analyzing how changes to weights affect the model's output on calibration data.
How GPTQ Works
GPTQ uses a layer-by-layer approach with the Optimal Brain Quantizer (OBQ) algorithm. For each layer:
- It processes a small calibration dataset through the model
- Calculates the Hessian matrix (sensitivity) for each weight
- Quantizes weights in order of their sensitivity
- Compensates for quantization errors to minimize quality loss
The result? You can compress a 7B parameter model from ~14GB (FP16) down to ~3.5GB (4-bit) with minimal perplexity increase—often less than 1 point.
Why Weight-Only?
Activations (the intermediate values during inference) are kept at FP16 because they're much smaller in volume and more sensitive to quantization. By focusing compression on weights—which dominate memory usage—GPTQ achieves maximum memory savings where it matters most.
Key Takeaway: GPTQ enables extreme compression (4-bit weights) for LLMs through careful, layer-wise quantization that preserves model quality while reducing memory requirements by up to 75%.