QAT for Activations vs Weights
What You'll Learn
You'll understand why quantization-aware training handles weights and activations differently, and the strategic implications of static versus dynamic quantization for each.
The Core Distinction
In QAT, weights and activations require fundamentally different quantization strategies because they behave differently during training and inference.
Weights are static parameters that change slowly through gradient updates. Their distribution is relatively stable, making them ideal for static quantization. During QAT, you insert fake quantization nodes that learn optimal scale and zero-point parameters for each weight tensor. These parameters converge alongside the weights themselves, allowing the network to adapt to quantization constraints.
Activations, however, are dynamic—their distributions vary with input data. For activations, you typically use dynamic range estimation during training. QAT collects statistics (min/max values) across batches to model the activation distributions. The network learns to produce activations that stay within quantizable ranges, but the actual quantization parameters may still adapt based on observed ranges.
The Practical Difference
Think of it like packing a suitcase (weights) versus dealing with weather (activations). You know exactly what clothes you have, so you can measure and pack them optimally. But weather changes—you need to prepare for a range of conditions.
During QAT:
- Weight quantization: Fixed scale/zero-point per tensor or channel, learned end-to-end
- Activation quantization: May use moving averages of observed ranges, requiring calibration-like statistics during training
This dual approach ensures weights are perfectly quantized while activations remain flexible enough to handle varying input distributions at inference time.
Key Takeaway: QAT treats weights with static quantization (learned fixed parameters) and activations with dynamic range tracking, reflecting their different stability characteristics during training and inference.