Learned Step Size Quantization (LSQ)
What you'll learn: How to make quantization step sizes learnable parameters that train alongside your network weights for better accuracy.
The Core Problem with Fixed Step Sizes
In standard quantization-aware training (QAT), we use fake quantization nodes with a scale parameter that determines how real values map to integers. Typically, this scale is computed from the weight or activation range and remains fixed or updates based on statistics—but it's not optimized through backpropagation like regular weights.
This is suboptimal. The quantization step size directly controls the discretization error, yet we're not letting the network learn the best step size for minimizing overall loss.
How LSQ Works
Learned Step Size Quantization treats the quantization scale (step size) as a learnable parameter that gets updated via gradient descent during training.
Instead of:
scale = compute_from_range(weights)
LSQ does:
scale = nn.Parameter(initial_scale) # Trainable!
During forward pass, values are quantized using this scale. During backward pass, gradients flow through the straight-through estimator (as usual for the quantized values), plus gradients update the scale parameter itself.
The network learns: "If I make the step size slightly smaller here, I reduce loss more than if I adjust the weights alone."
The Gradient Calculation
LSQ computes gradients with respect to the step size using the chain rule. The gradient measures how sensitive the loss is to changes in quantization granularity, allowing joint optimization of weights and quantization parameters.
Key Takeaway: LSQ treats quantization step sizes as first-class learnable parameters during QAT, allowing the network to discover optimal quantization granularity through backpropagation rather than relying on heuristic scale computations.