Course contentsShow
← AI Engineering
Lesson 1047 of 1,88625. Model Serving and Inference OptimizationFree lesson

Hardware Requirements for Quantized Models

Understanding GPU and CPU requirements for different quantization formats and inference frameworks.

Hardware Requirements for Quantized Models

You'll learn: How different quantization formats and inference frameworks map to specific GPU and CPU capabilities, helping you choose the right hardware for your deployment needs.

Why Hardware Matters for Quantization

Just because you've quantized a model to INT8 or INT4 doesn't mean any hardware can run it efficiently. Quantization reduces memory footprint, but to actually accelerate inference, your hardware needs specialized instructions to perform low-precision arithmetic fast.

Think of it like compressing a video: the file is smaller, but you need a device with the right codec support to play it smoothly. Without that codec, you might still play the video, but without speed benefits.

GPU Requirements by Quantization Format

FP16 quantization works on most modern GPUs (NVIDIA V100+, AMD MI series). It's widely supported and offers good speedups.

INT8 quantization requires Tensor Cores (NVIDIA Turing/Ampere+) or equivalent matrix acceleration hardware. Without these, INT8 operations fall back to slower FP32 paths, negating benefits.

INT4/2-bit formats need cutting-edge support: NVIDIA Ada (RTX 40-series) or Hopper (H100) GPUs with FP8/INT4 Tensor Cores. Older GPUs can load INT4 models but won't accelerate computation.

CPU Considerations

Modern CPUs (Intel AVX-512, ARM NEON) support INT8 operations efficiently. Tools like bitsandbytes and ONNX Runtime leverage these instructions. However, CPU inference remains significantly slower than GPU for large models—quantization just makes it less slow.

Framework-Specific Requirements

  • vLLM: Best with A100/H100 GPUs for GPTQ/AWQ support
  • TGI: Requires CUDA 11.8+ for INT8 kernels
  • TensorRT: Needs NVIDIA GPUs with appropriate compute capability (7.0+ for INT8)

Key Takeaway: Quantization saves memory on any hardware, but achieving faster inference requires matching your quantization format to hardware-accelerated instructions—check GPU compute capability and instruction set support before deployment.