Activation Quantization Challenges
What you'll learn: Why quantizing activations is harder than quantizing weights, especially when distributions are unbounded and non-uniform.
The Activation Problem
While weight quantization is relatively straightforward (weights are fixed after training and have known ranges), activation quantization presents unique challenges. Activations are computed dynamically during inference, and their distributions can vary dramatically:
- Unbounded ranges: Some activations (like ReLU outputs) theoretically have no upper limit
- Non-uniform distributions: Activations often contain outliers or follow skewed, heavy-tailed distributions
- Layer-dependent variability: Different layers produce wildly different activation patterns
Why This Matters for Extreme Low-Bit Quantization
In 8-bit quantization, you have 256 discrete levels to work with—enough to absorb some irregularity. But at 2-bit or 4-bit precision, you might have only 4 or 16 levels. Now those outliers become catastrophic:
Analogy: Imagine trying to represent temperatures using only 4 values. If 99% of readings are 20-30°C but one outlier is 1000°C, you must either:
- Clip the outlier (lose information)
- Expand your range to include it (waste precision on the common 20-30°C range)
Common Solutions
- Percentile clipping: Use calibration data to set min/max at 99th percentile, accepting minor clipping
- Per-channel quantization: Different scales for different output channels
- Learned clipping bounds: Train the network to adapt to quantization constraints (QAT)
- Mixed precision: Keep problematic layers in higher precision
Key Takeaway: Activation quantization is harder than weight quantization because activations have unbounded, non-uniform, data-dependent distributions that make finding optimal quantization parameters challenging, especially at extreme low-bit widths.