Course contentsShow
← Machine Learning and Deep Learning
Lesson 2639 of 3,53857. Model Compression: QuantizationFree lesson

PyTorch Dynamic Quantization

Implement dynamic quantization in PyTorch using torch.quantization for simple weight and activation quantization.

PyTorch Dynamic Quantization

What you'll learn: How to implement dynamic quantization in PyTorch to reduce model size and accelerate inference by quantizing weights and activations on-the-fly.

What is Dynamic Quantization?

Dynamic quantization is a straightforward quantization technique where weights are quantized ahead of time (converted to INT8 during model preparation), but activations are quantized dynamically at runtime, just before each operation. Think of it like packing your luggage efficiently (weights) before a trip, but deciding what to wear (activations) only when you need it.

Unlike static quantization, dynamic quantization doesn't require a calibration dataset because activation scales are computed on-the-fly based on the actual data flowing through the network.

Implementing in PyTorch

PyTorch makes dynamic quantization incredibly simple with torch.quantization.quantize_dynamic():

import torch
import torch.nn as nn

# Define a simple model
model = nn.Sequential(
    nn.Linear(128, 64),
    nn.ReLU(),
    nn.Linear(64, 10)
)

# Apply dynamic quantization to Linear layers
quantized_model = torch.quantization.quantize_dynamic(
    model,
    {nn.Linear},  # Specify which layer types to quantize
    dtype=torch.qint8  # Target quantization precision
)

# Use the quantized model for inference
x = torch.randn(1, 128)
output = quantized_model(x)

The function automatically converts specified layer types (typically nn.Linear, nn.LSTM, nn.GRU) to their dynamically quantized versions, where weights are stored as INT8 but computations adapt to each input batch.

When to Use Dynamic Quantization

Dynamic quantization works best for models where memory bandwidth is the bottleneck rather than compute—particularly in recurrent networks (LSTMs, GRUs) and large linear layers. It's the easiest quantization method to apply since it requires no calibration data or training modifications.

Key Takeaway: PyTorch dynamic quantization uses torch.quantization.quantize_dynamic() to automatically convert model weights to INT8 while quantizing activations on-the-fly, offering an easy path to faster inference without calibration datasets.