Course contentsShow
← Machine Learning and Deep Learning
Lesson 2641 of 3,53857. Model Compression: QuantizationFree lesson

Quantization of Specific Layer Types

Understand which layers benefit most from PTQ (linear, conv) and which are typically kept in full precision (batchnorm, softmax).

Quantization of Specific Layer Types

What you'll learn: You'll discover which neural network layers benefit most from post-training quantization and which should typically remain in full precision.

Why Layer Type Matters

Not all layers respond equally well to quantization. Some layer types are robust to precision reduction, while others are sensitive and can cause significant accuracy drops when quantized.

Quantization-Friendly Layers

Linear (Fully Connected) and Convolutional Layers are the best candidates for post-training quantization:

  • They contain the bulk of model parameters (weights)
  • Their operations are mathematically stable under reduced precision
  • They account for most computation time and memory usage
  • Weight distributions tend to be well-behaved and quantize cleanly

Think of these layers as workhorses performing repetitive multiplication-addition operations—they can tolerate some numerical "fuzziness" without breaking down.

Keep in Full Precision

Certain layers should typically stay in FP32:

  • Batch Normalization: Contains running statistics and scale/shift parameters that are sensitive to precision loss; often fused with preceding conv layers instead
  • Softmax: Requires high precision for stable exponential calculations and proper probability distributions
  • Layer Normalization: Similar sensitivity to batch norm
  • First and Last Layers: The input embedding and final classification layers often need higher precision to preserve accuracy

The Practical Pattern

A typical quantization strategy:

  1. Quantize all Conv2d and Linear layers to INT8
  2. Keep normalization and activation functions in FP32
  3. Monitor accuracy—if it drops significantly, selectively restore sensitive layers to FP32 (this is mixed precision quantization)

Key Takeaway: Linear and convolutional layers are prime quantization targets due to their computational weight and robustness, while normalization and probability-based layers typically need full precision to maintain model stability.