Per-Channel vs Per-Tensor Quantization
What you'll learn: You'll understand how choosing different granularities for quantization parameters affects both model accuracy and hardware performance.
The Core Trade-Off
When quantizing a neural network, you need to decide how many scale and zero-point parameters to use. This is the granularity question:
- Per-Tensor Quantization: Use one scale and one zero-point for an entire weight tensor or activation
- Per-Channel Quantization: Use separate scale/zero-point pairs for each output channel (filter) in a convolutional or linear layer
Think of it like adjusting exposure on a photo: per-tensor is like one brightness slider for the whole image, while per-channel gives you individual sliders for different regions.
Why Granularity Matters
Different channels in a layer often have vastly different dynamic ranges. Some filters might have weights clustered around 0.01, others around 1.0. With per-tensor quantization, you must choose parameters that work for all channels simultaneously, often wasting precision on some while clipping others.
Per-channel quantization adapts to each channel's unique distribution, preserving more information and typically reducing accuracy loss by 1-3% compared to per-tensor, especially in deeper networks.
The Hardware Consideration
The catch? Per-channel requires more computation and memory at inference time. Instead of one multiply-add for dequantization, you need per-channel operations. Many accelerators support per-channel for weights (computed once) but prefer per-tensor for activations (computed repeatedly).
This is why weights commonly use per-channel while activations use per-tensor in production systems—it's the sweet spot balancing accuracy and efficiency.
Key Takeaway: Per-channel quantization offers better accuracy by adapting to individual channel distributions but increases computational overhead; per-tensor is faster but less precise, making the choice a hardware-accuracy trade-off.