Course contentsShow
← Machine Learning and Deep Learning
Lesson 2959 of 3,53864. GPU Inference OptimizationFree lesson

Layer and Tensor Fusion

How TensorRT combines multiple layers into single GPU kernels to reduce memory bandwidth and kernel launch overhead.

Layer and Tensor Fusion

What You'll Learn

You'll learn how TensorRT combines multiple neural network layers into single GPU kernels to dramatically reduce memory traffic and kernel launch overhead.

The Core Idea

When a neural network runs on a GPU, each layer typically executes as a separate kernel (a GPU function). Between kernels, intermediate results must be written to and read from GPU memory. This memory traffic is expensive—often the bottleneck in inference performance.

Layer fusion solves this by merging multiple operations into a single kernel. Instead of:

  1. Run Conv → write to memory
  2. Read from memory → run BatchNorm → write to memory
  3. Read from memory → run ReLU → write to memory

TensorRT creates:

  1. Run Conv+BatchNorm+ReLU in one kernel → write final result to memory

Why This Matters

Memory bandwidth savings: Intermediate tensors never leave GPU registers, eliminating expensive DRAM round-trips.

Kernel launch reduction: Each kernel launch has overhead (~5-20 microseconds). Fusing 3 layers means 3× fewer launches.

Better cache utilization: Data stays hot in L1/L2 cache throughout the fused computation.

Common Fusion Patterns

  • Vertical fusion: Sequential operations (Conv → BN → ReLU)
  • Horizontal fusion: Independent operations at the same depth
  • Tensor fusion: Combining operations on the same tensor (element-wise ops)

TensorRT's optimization pipeline (which you learned in the previous lesson) automatically identifies these patterns during graph optimization and generates optimized, fused kernels tailored to your model's structure.

Key Takeaway: Layer and tensor fusion eliminates intermediate memory operations by combining multiple layers into single GPU kernels, often yielding 2-5× speedups by reducing memory bandwidth bottlenecks and kernel launch overhead.