Course contentsShow
← Machine Learning and Deep Learning
Lesson 2628 of 3,53857. Model Compression: QuantizationFree lesson

Where to Apply Quantization in a Model

Which layers and operations to quantize: weights, activations, gradients, and why some layers are sensitive.

Where to Apply Quantization in a Model

What you'll learn: You'll discover which specific parts of a neural network can be quantized and why some layers are more sensitive to quantization than others.

The Landscape of Quantizable Components

Not all parts of a neural network benefit equally from quantization. You can quantize three main components:

Weights

These are the learned parameters stored in memory. Quantizing weights reduces model size significantly and is usually well-tolerated because weights are relatively stable after training. Most quantization strategies start here.

Activations

These are the intermediate outputs flowing through the network during inference. Quantizing activations speeds up computation but requires careful handling because their dynamic range varies across layers and inputs.

Gradients

During training, you can also quantize gradients to reduce communication overhead in distributed training. However, this is the most sensitive operation—aggressive gradient quantization can destabilize training.

Layer Sensitivity Matters

Not all layers respond equally to quantization:

  • Early layers (close to input) are often more sensitive because they extract low-level features where precision matters
  • Final classification layers are sensitive because small changes in logits can flip predictions
  • Batch normalization and layer normalization layers are typically kept in higher precision
  • Large convolutional or linear layers in the middle are prime candidates—they're robust and contain most parameters

Think of it like a recipe: You can round 2.5 cups of flour to 2 or 3 cups without disaster, but rounding 0.25 teaspoons of salt to 0 or 1 teaspoon ruins the dish. Precision matters more in some places.

Key Takeaway: Start by quantizing weights in middle layers, proceed cautiously with activations, keep sensitive layers (first, last, normalization) in higher precision, and only quantize gradients if you need distributed training efficiency and can afford careful tuning.