Dynamic vs Static Shape Optimization
What you'll learn: How input shape variability affects inference performance and when to apply padding or bucketing strategies to optimize your models.
The Core Problem
When running inference with pre-trained models, your inputs won't always be the same size. A text classifier might receive 10 tokens in one request and 500 in another. Image models might get photos of varying dimensions. This variability creates a choice: optimize for dynamic shapes (any size, anytime) or static shapes (fixed, predictable sizes).
Static Shape Optimization
Think of static shapes like assembly line manufacturing—everything's the same size, so the process runs smoothly and predictably. When your model expects exactly 512 tokens every time, modern optimization techniques (like torch.compile() and ONNX Runtime) can pre-calculate operations and allocate memory efficiently. This delivers maximum speed but requires your inputs to match that exact shape.
Strategy: Use padding to force all inputs to a fixed length. A 50-token sentence gets padded to 512; a 600-token document gets truncated to 512.
Dynamic Shape Optimization
Dynamic shapes are like a flexible factory—they handle any size but with some overhead cost. The model must adapt its computation graph for each unique input shape, which slows down highly optimized inference engines.
Strategy: Use bucketing—group similar-sized inputs together (e.g., 0-128, 129-256, 257-512 tokens). Process each bucket with its own optimized static shape. This balances flexibility with performance, minimizing wasted padding while maintaining optimization benefits.
When to Choose Each
Use static shapes when input sizes are predictable and you need maximum speed. Use bucketing when inputs vary widely but you still want good performance. Use fully dynamic shapes only when flexibility matters more than speed.
Key Takeaway: Static shapes enable the fastest inference through aggressive optimization, while bucketing provides a practical middle ground for variable-length inputs without sacrificing too much performance.