Course contentsShow
← Machine Learning and Deep Learning
Lesson 1357 of 3,53830. Vision TransformersFree lesson

Patch Merging as Downsampling

How concatenating and projecting neighboring patches reduces spatial dimensions while expanding channels.

Patch Merging as Downsampling

What you'll learn: How Swin Transformer reduces spatial resolution while increasing feature channels through patch merging.

The Core Idea

In CNNs, we use pooling or strided convolutions to create pyramidal feature hierarchies—progressively smaller spatial dimensions with richer feature representations. Swin Transformer achieves the same goal using patch merging, a simple but clever downsampling operation.

How Patch Merging Works

Imagine you have a grid of image patches, each represented by a feature vector. To downsample:

  1. Concatenate neighboring patches: Group each 2×2 neighborhood of patches together and concatenate their features
  2. Linear projection: Apply a learned linear layer to reduce the expanded dimension

Analogy: Think of combining four adjacent puzzle pieces into one larger piece. Instead of discarding information (like max pooling), you merge all the details and compress them into a denser representation.

The Dimension Dance

If you start with patches of dimension C arranged in an H × W grid:

  • After concatenating 2×2 neighbors: each merged patch has 4C channels
  • Spatial dimensions halve: H/2 × W/2 grid
  • After linear projection (typically to 2C): you get 2C channels at H/2 × W/2 resolution

This mimics CNN behavior: as you go deeper, spatial resolution ↓, feature richness (channels) ↑.

Why This Matters

Patch merging allows Swin Transformer to process images at multiple scales efficiently, making it suitable for dense prediction tasks like object detection and segmentation—areas where vanilla ViT struggles due to its fixed single-resolution representation.

Key Takeaway: Patch merging downsamples spatial dimensions by concatenating neighboring patch features and projecting them to a higher channel count, creating hierarchical representations similar to CNNs while preserving information through learned transformations.