QAT in PyTorch
What You'll Learn
You'll learn how to implement Quantization-Aware Training in PyTorch using the built-in torch.quantization module, covering model preparation, training with fake quantization, and final conversion.
Core Idea
PyTorch provides a straightforward workflow for QAT through three main steps: prepare, train, and convert. Instead of manually inserting fake quantization nodes, PyTorch automates this process while giving you control over which layers get quantized and how.
The Three-Step QAT Workflow
1. Prepare the Model with prepare_qat
First, configure your model with a QConfig that specifies quantization settings (like per-channel or per-tensor), then call prepare_qat. This function inserts fake quantization nodes for both weights and activations, essentially creating a "quantization-aware" version of your model.
import torch.quantization as quantization
# Configure quantization
model.qconfig = quantization.get_default_qat_qconfig('fbgemm')
# Insert fake quant nodes
model_prepared = quantization.prepare_qat(model, inplace=False)
2. Train with Fake Quantization
Train your prepared model normally. During forward passes, fake quantization simulates quantization effects (rounding, clipping) while gradients flow through using the straight-through estimator during backpropagation. The model learns to be robust to quantization noise.
3. Convert to Quantized Model
After training, call convert to replace fake quantization operations with actual integer ops:
model_quantized = quantization.convert(model_prepared, inplace=False)
Now your model runs with real INT8 operations, delivering actual speedup and size reduction.
Analogy
Think of prepare_qat as putting training wheels on a bicycle (fake quant), training as learning to balance with those wheels on, and convert as removing the training wheels—you're now riding with the real thing.
Key Takeaway: PyTorch QAT uses prepare_qat to insert fake quantization, trains the model to adapt to quantization constraints, then convert produces the final INT8 model ready for efficient deployment.