Batch Normalization Folding in QAT
You'll learn why and how to fold batch normalization layers into adjacent convolution or linear layers before starting quantization-aware training.
The Problem with Batch Norm in Quantization
Batch normalization adds extra computation and complicates quantization. During inference, batch norm applies a learned affine transformation: output = gamma * (input - mean) / sqrt(variance + epsilon) + beta. This involves floating-point operations that we'd prefer to eliminate when quantizing.
Here's the key insight: batch norm's inference-time behavior is mathematically equivalent to adjusting the weights and biases of the preceding layer.
How Folding Works
For a convolution followed by batch norm:
- The conv produces:
y = W * x + b - Batch norm transforms it:
output = gamma * (y - mean) / sqrt(var) + beta
We can fold this into the conv by computing new weights and biases:
W_folded = W * gamma / sqrt(var + epsilon)
b_folded = gamma * (b - mean) / sqrt(var + epsilon) + beta
Now the conv alone produces the same output as conv + batch norm together, and we've eliminated the batch norm layer entirely.
Why This Matters for QAT
By folding batch norm before QAT begins:
- You quantize fewer layers (no batch norm to worry about)
- The fake quantization nodes operate on simpler computational graphs
- Scale and zero-point parameters are learned for the true inference architecture
- The final quantized model matches what you trained
Key Takeaway: Batch normalization folding merges batch norm parameters into preceding layers before QAT, creating a simpler, equivalent network that's easier to quantize accurately and matches your deployment architecture exactly.