Pyramid Vision Transformer (PVT)
What You'll Learn
You'll discover how PVT creates a multi-scale feature hierarchy using spatial-reduction attention, offering an alternative to Swin's shifted windows approach.
The Core Idea
While Swin Transformer builds hierarchies through window partitioning and patch merging, Pyramid Vision Transformer (PVT) takes a different route: it progressively reduces the spatial dimensions of feature maps using spatial-reduction attention at each stage.
Think of it like looking at a photograph through increasingly zoomed-out lenses. Each stage of PVT processes a lower-resolution version of the features, capturing patterns at different scales—fine details early on, broader context later.
How Spatial-Reduction Attention Works
Instead of dividing images into fixed windows, PVT applies a spatial reduction operation to the keys and values in self-attention before computing attention scores. This means:
- Queries stay at full resolution for the current stage
- Keys and values are downsampled (e.g., by 2× or 4×) using convolution or pooling
- Attention is computed between full-resolution queries and reduced-resolution keys/values
This dramatically reduces computational cost—if you cut spatial dimensions in half for K and V, you reduce attention complexity by roughly 4×.
Building the Pyramid
PVT stacks multiple stages, each with:
- Smaller patch embeddings (progressively merging patches)
- Spatial-reduction ratios that increase computational efficiency
- Feature maps that shrink in height/width but grow in channels
This creates a pyramid of features similar to CNNs like ResNet—perfect for dense prediction tasks like segmentation and detection.
Key Takeaway: PVT builds hierarchical vision features by progressively downsampling spatial dimensions through spatial-reduction attention, balancing efficiency and multi-scale representation without relying on window partitioning.