Course contentsShow
← Machine Learning and Deep Learning
Lesson 2660 of 3,53857. Model Compression: QuantizationFree lesson

Per-Channel vs Per-Tensor Quantization

Comparing granularity of quantization parameters and their impact on model quality and hardware efficiency.

Per-Channel vs Per-Tensor Quantization

What you'll learn: You'll understand how choosing different granularities for quantization parameters affects both model accuracy and hardware performance.

The Core Trade-Off

When quantizing a neural network, you need to decide how many scale and zero-point parameters to use. This is the granularity question:

  • Per-Tensor Quantization: Use one scale and one zero-point for an entire weight tensor or activation
  • Per-Channel Quantization: Use separate scale/zero-point pairs for each output channel (filter) in a convolutional or linear layer

Think of it like adjusting exposure on a photo: per-tensor is like one brightness slider for the whole image, while per-channel gives you individual sliders for different regions.

Why Granularity Matters

Different channels in a layer often have vastly different dynamic ranges. Some filters might have weights clustered around 0.01, others around 1.0. With per-tensor quantization, you must choose parameters that work for all channels simultaneously, often wasting precision on some while clipping others.

Per-channel quantization adapts to each channel's unique distribution, preserving more information and typically reducing accuracy loss by 1-3% compared to per-tensor, especially in deeper networks.

The Hardware Consideration

The catch? Per-channel requires more computation and memory at inference time. Instead of one multiply-add for dequantization, you need per-channel operations. Many accelerators support per-channel for weights (computed once) but prefer per-tensor for activations (computed repeatedly).

This is why weights commonly use per-channel while activations use per-tensor in production systems—it's the sweet spot balancing accuracy and efficiency.

Key Takeaway: Per-channel quantization offers better accuracy by adapting to individual channel distributions but increases computational overhead; per-tensor is faster but less precise, making the choice a hardware-accuracy trade-off.