Dynamic Shapes and Optimization Profiles
You'll learn how TensorRT handles variable input sizes using optimization profiles to maintain high performance across different input dimensions.
The Variable Input Challenge
Most real-world models need to accept inputs of varying sizes—images with different resolutions, text sequences of different lengths, or batch sizes that change per request. Traditional static graph optimization assumes fixed dimensions, but TensorRT's dynamic shapes feature lets you build engines that adapt to variable inputs without sacrificing too much performance.
What Are Optimization Profiles?
An optimization profile tells TensorRT the range of input dimensions you expect. You define three key points for each dynamic dimension:
- Minimum shape: smallest input size you'll use
- Optimal shape: most common size (TensorRT optimizes heavily for this)
- Maximum shape: largest input you'll ever send
TensorRT then generates kernels and selects strategies that work across this entire range, with peak performance near your optimal shape.
How It Works
When you build a TensorRT engine with dynamic shapes, you're essentially saying: "My batch size varies from 1 to 32, and my sequence length varies from 10 to 512." TensorRT pre-computes optimized execution plans for representative points in that space. At inference time, the engine adapts to your actual input size using these plans.
Think of it like a restaurant preparing for variable party sizes: they know most groups are 2-4 people (optimal), but can handle 1-8 (min-max), so they arrange tables and staff accordingly.
Key Takeaway: TensorRT's optimization profiles let you handle variable input dimensions efficiently by defining min/optimal/max ranges, enabling flexible inference while maintaining near-optimal performance for your most common input shapes.