Static vs Dynamic Shape Handling
What You'll Learn
You'll understand the critical trade-off between optimizing models for fixed input shapes versus maintaining flexibility for variable-sized inputs during GPU inference.
The Core Problem
When serving models on GPUs, you face a fundamental choice: static shapes (fixed input dimensions) or dynamic shapes (variable input dimensions). This decision profoundly impacts both performance and flexibility.
Static shape handling means your model is compiled and optimized assuming inputs always have the same dimensions—for example, images always 224×224 or sequences always length 512. The compiler can pre-allocate memory, fuse operations aggressively, and eliminate conditional branches. Think of it like a factory assembly line built for one specific product size: incredibly efficient but inflexible.
Dynamic shape handling accommodates variable inputs—different image sizes, varying sequence lengths, or batch sizes that change per request. The runtime must handle conditional logic, recompile kernels, or use less aggressive optimizations. It's like a flexible manufacturing system: versatile but with overhead.
The Trade-offs
Static advantages:
- Maximum kernel fusion and optimization
- Predictable memory allocation
- Fastest single-request latency
- Better for uniform workloads
Dynamic advantages:
- Handles real-world variable inputs naturally
- No padding waste for sequences/batches
- Essential for dynamic batching strategies
- Simpler preprocessing pipelines
In formats like ONNX and TorchScript, you choose at export time. Static shapes unlock ONNX Runtime's most aggressive graph optimizations. Dynamic shapes require runtime shape inference but handle diverse workloads without rejection or padding overhead.
Practical pattern: Use static shapes when input distributions are uniform (e.g., fixed-resolution image classification). Use dynamic shapes when variability is inherent (e.g., text generation, object detection with varying image sizes).
Key Takeaway: Static shape handling maximizes GPU performance through aggressive compile-time optimization, while dynamic shapes maintain serving flexibility at the cost of some runtime overhead—choose based on whether your workload uniformity justifies the optimization gains.