Course contentsShow
← Machine Learning and Deep Learning
Lesson 1358 of 3,53830. Vision TransformersFree lesson

Pyramid Vision Transformer (PVT)

Alternative hierarchical design using spatial-reduction attention to progressively downsample feature maps.

Pyramid Vision Transformer (PVT)

What You'll Learn

You'll discover how PVT creates a multi-scale feature hierarchy using spatial-reduction attention, offering an alternative to Swin's shifted windows approach.

The Core Idea

While Swin Transformer builds hierarchies through window partitioning and patch merging, Pyramid Vision Transformer (PVT) takes a different route: it progressively reduces the spatial dimensions of feature maps using spatial-reduction attention at each stage.

Think of it like looking at a photograph through increasingly zoomed-out lenses. Each stage of PVT processes a lower-resolution version of the features, capturing patterns at different scales—fine details early on, broader context later.

How Spatial-Reduction Attention Works

Instead of dividing images into fixed windows, PVT applies a spatial reduction operation to the keys and values in self-attention before computing attention scores. This means:

  1. Queries stay at full resolution for the current stage
  2. Keys and values are downsampled (e.g., by 2× or 4×) using convolution or pooling
  3. Attention is computed between full-resolution queries and reduced-resolution keys/values

This dramatically reduces computational cost—if you cut spatial dimensions in half for K and V, you reduce attention complexity by roughly 4×.

Building the Pyramid

PVT stacks multiple stages, each with:

  • Smaller patch embeddings (progressively merging patches)
  • Spatial-reduction ratios that increase computational efficiency
  • Feature maps that shrink in height/width but grow in channels

This creates a pyramid of features similar to CNNs like ResNet—perfect for dense prediction tasks like segmentation and detection.

Key Takeaway: PVT builds hierarchical vision features by progressively downsampling spatial dimensions through spatial-reduction attention, balancing efficiency and multi-scale representation without relying on window partitioning.