Per-Tensor vs Per-Channel Quantization
What You'll Learn
You'll understand the trade-off between using a single scale factor for an entire tensor versus individual scale factors per output channel, and how this choice affects accuracy preservation.
The Core Concept
When quantizing weights or activations, you need to choose the granularity of your quantization parameters (scale and zero-point). There are two main approaches:
Per-Tensor Quantization uses a single scale factor and zero-point for the entire tensor. Think of it like adjusting the volume on a single master knob for all instruments in an orchestra—simple, but some instruments may be too loud or too quiet.
Per-Channel Quantization uses separate scale factors and zero-points for each output channel (typically applied to weight tensors). This is like having individual volume controls for each instrument—more complex, but each gets optimal treatment.
The Trade-off
Per-Tensor:
- ✅ Simpler implementation
- ✅ Lower memory overhead (fewer quantization parameters)
- ✅ Faster computation
- ❌ Coarser representation—channels with different value ranges suffer
Per-Channel:
- ✅ Better accuracy preservation (especially when channels have very different ranges)
- ✅ Each channel optimally uses its quantization range
- ❌ More quantization parameters to store
- ❌ Slightly more complex hardware implementation
When It Matters
Per-channel quantization shines when different output channels in a convolutional or linear layer have dramatically different weight distributions. Some channels might have weights ranging from -0.01 to 0.01, while others span -2.0 to 2.0. Per-tensor would force them to share the same scale, wasting precious quantization bits on the smaller-range channel.
Key Takeaway: Per-channel quantization trades modest storage overhead for better accuracy by respecting the unique value distributions of each output channel, making it the preferred choice for weight quantization in most production systems.