Course contentsShow
← Machine Learning and Deep Learning
Lesson 2649 of 3,53857. Model Compression: QuantizationFree lesson

Batch Normalization Folding in QAT

Why and how to fold batch norm into preceding layers before quantization-aware training begins.

Batch Normalization Folding in QAT

You'll learn why and how to fold batch normalization layers into adjacent convolution or linear layers before starting quantization-aware training.

The Problem with Batch Norm in Quantization

Batch normalization adds extra computation and complicates quantization. During inference, batch norm applies a learned affine transformation: output = gamma * (input - mean) / sqrt(variance + epsilon) + beta. This involves floating-point operations that we'd prefer to eliminate when quantizing.

Here's the key insight: batch norm's inference-time behavior is mathematically equivalent to adjusting the weights and biases of the preceding layer.

How Folding Works

For a convolution followed by batch norm:

  1. The conv produces: y = W * x + b
  2. Batch norm transforms it: output = gamma * (y - mean) / sqrt(var) + beta

We can fold this into the conv by computing new weights and biases:

W_folded = W * gamma / sqrt(var + epsilon)
b_folded = gamma * (b - mean) / sqrt(var + epsilon) + beta

Now the conv alone produces the same output as conv + batch norm together, and we've eliminated the batch norm layer entirely.

Why This Matters for QAT

By folding batch norm before QAT begins:

  • You quantize fewer layers (no batch norm to worry about)
  • The fake quantization nodes operate on simpler computational graphs
  • Scale and zero-point parameters are learned for the true inference architecture
  • The final quantized model matches what you trained

Key Takeaway: Batch normalization folding merges batch norm parameters into preceding layers before QAT, creating a simpler, equivalent network that's easier to quantize accurately and matches your deployment architecture exactly.