Course contentsShow
← Machine Learning and Deep Learning
Lesson 2654 of 3,53857. Model Compression: QuantizationFree lesson

QAT Best Practices and Pitfalls

Common issues: gradient explosion, calibration data selection, avoiding accuracy degradation, and QAT convergence.

QAT Best Practices and Pitfalls

What you'll learn: How to avoid common mistakes when applying Quantization-Aware Training and ensure your model converges smoothly without accuracy loss.

Why QAT Can Go Wrong

QAT introduces fake quantization nodes and the straight-through estimator during training, which fundamentally changes gradient flow. Without care, you can encounter issues that don't exist in standard training: gradient explosions, poor convergence, or unexpected accuracy drops.

Common Pitfalls and Solutions

Gradient Explosion

The straight-through estimator treats discrete rounding as identity during backpropagation. Combined with quantization clipping, gradients can behave erratically, especially early in training.

Best practice: Start with a lower learning rate (often 10-100× smaller than pre-training). Use gradient clipping if needed. Consider freezing batch normalization statistics initially.

Poor Calibration Data Selection

Your calibration dataset determines the scale and zero-point parameters for activations. Using non-representative data leads to severe clipping and accuracy loss.

Best practice: Use diverse, representative samples from your training set (typically 100-1000 images). Ensure all major data distributions are covered—avoid just the first batch!

Accuracy Degradation

Not all layers tolerate quantization equally. Some operations (like depth-wise convolutions or early layers) are particularly sensitive.

Best practice: Use mixed-precision QAT selectively. Keep sensitive layers at higher precision (INT16 or FP16). Monitor per-layer quantization error during training.

QAT Convergence Issues

Starting QAT from a randomly initialized model rarely works. The model must learn representations while adapting to quantization noise.

Best practice: Always start from a pre-trained floating-point model and fine-tune with QAT. Train for fewer epochs (typically 10-20% of original training). Use a learning rate schedule that decays smoothly.

Key Takeaway: QAT success depends on lower learning rates, representative calibration data, selective mixed-precision strategies, and fine-tuning from pre-trained models—treat it as careful fine-tuning, not fresh training.