Course contentsShow
← AI Engineering
Lesson 71 of 1,8862. Working with Pre-trained ModelsFree lesson

Dynamic vs Static Shape Optimization

How input shape variability affects performance and when to use padding or bucketing strategies.

Dynamic vs Static Shape Optimization

What you'll learn: How input shape variability affects inference performance and when to apply padding or bucketing strategies to optimize your models.

The Core Problem

When running inference with pre-trained models, your inputs won't always be the same size. A text classifier might receive 10 tokens in one request and 500 in another. Image models might get photos of varying dimensions. This variability creates a choice: optimize for dynamic shapes (any size, anytime) or static shapes (fixed, predictable sizes).

Static Shape Optimization

Think of static shapes like assembly line manufacturing—everything's the same size, so the process runs smoothly and predictably. When your model expects exactly 512 tokens every time, modern optimization techniques (like torch.compile() and ONNX Runtime) can pre-calculate operations and allocate memory efficiently. This delivers maximum speed but requires your inputs to match that exact shape.

Strategy: Use padding to force all inputs to a fixed length. A 50-token sentence gets padded to 512; a 600-token document gets truncated to 512.

Dynamic Shape Optimization

Dynamic shapes are like a flexible factory—they handle any size but with some overhead cost. The model must adapt its computation graph for each unique input shape, which slows down highly optimized inference engines.

Strategy: Use bucketing—group similar-sized inputs together (e.g., 0-128, 129-256, 257-512 tokens). Process each bucket with its own optimized static shape. This balances flexibility with performance, minimizing wasted padding while maintaining optimization benefits.

When to Choose Each

Use static shapes when input sizes are predictable and you need maximum speed. Use bucketing when inputs vary widely but you still want good performance. Use fully dynamic shapes only when flexibility matters more than speed.

Key Takeaway: Static shapes enable the fastest inference through aggressive optimization, while bucketing provides a practical middle ground for variable-length inputs without sacrificing too much performance.