Course contentsShow
← Machine Learning and Deep Learning
Lesson 2647 of 3,53857. Model Compression: QuantizationFree lesson

Learning Scale and Zero-Point Parameters

Making quantization parameters trainable to find optimal ranges for each layer during QAT.

Learning Scale and Zero-Point Parameters

What you'll learn: How to make quantization parameters trainable during QAT to automatically discover the best quantization ranges for each layer.

The Core Idea

In previous lessons, you learned about scale and zero-point parameters that control how floating-point values map to integers during quantization. Until now, these parameters were typically computed using calibration methods (like MinMax or entropy-based approaches).

But here's the key insight: we can treat scale and zero-point as learnable parameters during Quantization-Aware Training (QAT), just like weights and biases!

Why Make Them Learnable?

Think of calibration methods as using a fixed ruler to measure all your data. Sometimes that ruler isn't optimal for every layer—some layers might benefit from a different measurement range.

By making scale and zero-point learnable:

  • Each layer can adapt its quantization range during training
  • The model learns which values are most important to preserve precisely
  • Quantization becomes part of the optimization problem, not just a post-processing step

How It Works in Practice

During QAT, the fake quantization nodes now have two additional parameters that receive gradients:

  1. Scale (s) – determines the step size between quantized values
  2. Zero-point (z) – shifts the quantization range asymmetrically

These parameters are updated via backpropagation alongside model weights. The Straight-Through Estimator still handles the non-differentiable rounding operation for activations and weights, but scale and zero-point get "real" gradients based on the quantization loss.

The optimizer can now discover, for example, that a particular convolutional layer benefits from a tighter quantization range, while another needs broader coverage.

Key Takeaway: Making quantization scale and zero-point parameters learnable during QAT allows the model to automatically find optimal quantization ranges per layer, reducing accuracy loss compared to fixed calibration approaches.