Dynamic vs Static Quantization
You'll learn: The key differences between dynamic quantization (which quantizes activations on-the-fly during inference) and static quantization (which pre-calibrates activation ranges using sample data).
The Core Difference
After deciding to apply post-training quantization, you face a choice about when and how to quantize activations:
Dynamic Quantization quantizes weights ahead of time but converts activations from floating point to integers during runtime for each inference. The scale and zero-point for activations are computed on-the-fly based on the actual data flowing through the network.
Static Quantization goes further: it quantizes both weights and activations beforehand. This requires a calibration step where you run representative data through your model to observe the typical range of activation values. These observed ranges determine fixed quantization parameters that are baked into the model.
The Trade-Off
Think of it like restaurant preparation:
- Dynamic quantization is like a chef who preps ingredients (weights) in advance but adjusts seasoning (activation quantization) to taste while cooking each dish.
- Static quantization is like a standardized recipe where every measurement is predetermined, including seasonings—faster execution but requires testing batches first to get it right.
Dynamic offers easier setup (no calibration data needed) but slightly slower inference since quantization happens at runtime. Static delivers maximum speed and efficiency but requires calibration data and careful tuning to avoid accuracy loss from fixed activation ranges.
Key Takeaway: Dynamic quantization quantizes activations during each forward pass, while static quantization pre-determines activation quantization parameters through calibration—static is faster but requires more setup effort.