Learning Scale and Zero-Point Parameters
What you'll learn: How to make quantization parameters trainable during QAT to automatically discover the best quantization ranges for each layer.
The Core Idea
In previous lessons, you learned about scale and zero-point parameters that control how floating-point values map to integers during quantization. Until now, these parameters were typically computed using calibration methods (like MinMax or entropy-based approaches).
But here's the key insight: we can treat scale and zero-point as learnable parameters during Quantization-Aware Training (QAT), just like weights and biases!
Why Make Them Learnable?
Think of calibration methods as using a fixed ruler to measure all your data. Sometimes that ruler isn't optimal for every layer—some layers might benefit from a different measurement range.
By making scale and zero-point learnable:
- Each layer can adapt its quantization range during training
- The model learns which values are most important to preserve precisely
- Quantization becomes part of the optimization problem, not just a post-processing step
How It Works in Practice
During QAT, the fake quantization nodes now have two additional parameters that receive gradients:
- Scale (
s) – determines the step size between quantized values - Zero-point (
z) – shifts the quantization range asymmetrically
These parameters are updated via backpropagation alongside model weights. The Straight-Through Estimator still handles the non-differentiable rounding operation for activations and weights, but scale and zero-point get "real" gradients based on the quantization loss.
The optimizer can now discover, for example, that a particular convolutional layer benefits from a tighter quantization range, while another needs broader coverage.
Key Takeaway: Making quantization scale and zero-point parameters learnable during QAT allows the model to automatically find optimal quantization ranges per layer, reducing accuracy loss compared to fixed calibration approaches.