Course contentsShow
← Machine Learning and Deep Learning
Lesson 2633 of 3,53857. Model Compression: QuantizationFree lesson

Weight-Only Quantization

Understand quantizing only weights while keeping activations in full precision, commonly used for inference memory reduction.

Weight-Only Quantization

What You'll Learn

You'll learn how to reduce model memory by quantizing only the weights while keeping activations in full precision—a simple yet effective compression technique.

The Core Idea

Weight-only quantization is a selective approach where you convert model weights (the learned parameters) from 32-bit floating point to lower precision (typically 8-bit integers), but leave activations at full precision during inference.

Think of it like compressing a recipe book: you simplify the ingredient measurements (weights) to save space, but when cooking (running inference), you still measure the actual ingredients (activations) precisely. This asymmetric approach gives you immediate memory savings without the complexity of calibrating activation quantization.

Why Weight-Only?

This strategy is particularly attractive because:

  1. Weights are static after training—they don't change during inference, making them safe to quantize once
  2. Activations vary with each input, making them trickier to quantize well
  3. You get ~75% memory reduction (FP32→INT8) with minimal accuracy loss
  4. Simpler implementation—no need for calibration datasets or profiling activation ranges

The Trade-off

While weights live in INT8 in memory, they're dequantized to FP32 just before computation. This means:

  • ✅ Memory savings: Models load faster, fit on smaller devices
  • ✅ Easier deployment: No special runtime requirements
  • ❌ Limited speed gains: Computation still happens in FP32, so inference isn't as fast as full INT8 quantization

Weight-only quantization shines when memory—not computation speed—is your bottleneck, such as loading large language models on consumer hardware.

Key Takeaway: Weight-only quantization compresses model storage by 4× while keeping inference logic simple, trading maximum speed for ease of implementation and deployment.