Course contentsShow
← Machine Learning and Deep Learning
Lesson 2951 of 3,53864. GPU Inference OptimizationFree lesson

Operator Fusion in Graph Optimization

Understand how consecutive operations like conv-batchnorm-relu are fused into single kernels for speed.

Operator Fusion in Graph Optimization

What you'll learn: How consecutive operations like conv-batchnorm-relu are fused into single GPU kernels to dramatically speed up inference.

The Problem with Sequential Operations

When you run a neural network, operations like convolution, batch normalization, and ReLU activation typically execute as separate GPU kernels. Each kernel launch incurs overhead: memory reads/writes, kernel launch latency, and intermediate data storage. For a conv → batchnorm → relu sequence, data must be written to GPU memory after convolution, read back for batchnorm, written again, then read once more for ReLU.

What Operator Fusion Does

Operator fusion combines multiple consecutive operations into a single GPU kernel. Instead of three separate kernels, you get one fused kernel that:

  • Reads input data once
  • Applies convolution, batchnorm, and ReLU computations in sequence
  • Writes the final result once

Think of it like cooking: instead of washing dishes between chopping, sautéing, and seasoning (three separate tasks with cleanup), you do all three in one continuous workflow at the same workstation.

Why It Matters for Inference

Fused operators eliminate redundant memory transfers—the slowest part of GPU computation. This is especially valuable for inference where:

  • Memory bandwidth is often the bottleneck
  • Latency matters more than flexibility
  • The computation graph is static and known ahead of time

Graph optimization frameworks (ONNX Runtime, TorchScript) automatically identify fusion opportunities and rewrite your model's computational graph during conversion or compilation.

Key Takeaway: Operator fusion merges consecutive operations into single GPU kernels, eliminating costly intermediate memory transfers and dramatically improving inference speed—a core optimization in tools like ONNX Runtime and TorchScript.