Patch Merging as Downsampling
What you'll learn: How Swin Transformer reduces spatial resolution while increasing feature channels through patch merging.
The Core Idea
In CNNs, we use pooling or strided convolutions to create pyramidal feature hierarchies—progressively smaller spatial dimensions with richer feature representations. Swin Transformer achieves the same goal using patch merging, a simple but clever downsampling operation.
How Patch Merging Works
Imagine you have a grid of image patches, each represented by a feature vector. To downsample:
- Concatenate neighboring patches: Group each 2×2 neighborhood of patches together and concatenate their features
- Linear projection: Apply a learned linear layer to reduce the expanded dimension
Analogy: Think of combining four adjacent puzzle pieces into one larger piece. Instead of discarding information (like max pooling), you merge all the details and compress them into a denser representation.
The Dimension Dance
If you start with patches of dimension C arranged in an H × W grid:
- After concatenating 2×2 neighbors: each merged patch has 4C channels
- Spatial dimensions halve: H/2 × W/2 grid
- After linear projection (typically to 2C): you get 2C channels at H/2 × W/2 resolution
This mimics CNN behavior: as you go deeper, spatial resolution ↓, feature richness (channels) ↑.
Why This Matters
Patch merging allows Swin Transformer to process images at multiple scales efficiently, making it suitable for dense prediction tasks like object detection and segmentation—areas where vanilla ViT struggles due to its fixed single-resolution representation.
Key Takeaway: Patch merging downsamples spatial dimensions by concatenating neighboring patch features and projecting them to a higher channel count, creating hierarchical representations similar to CNNs while preserving information through learned transformations.