Course contentsShow
← Machine Learning and Deep Learning
Lesson 1353 of 3,53830. Vision TransformersFree lesson

Swin Transformer: Shifted Windows

Introduction to Swin Transformer's shifted window attention mechanism for efficient local-global modeling.

Swin Transformer: Shifted Windows

What you'll learn: How Swin Transformer uses a clever "shifted window" mechanism to capture both local and global image relationships efficiently.

The Problem with Global Attention

Vanilla Vision Transformers (ViT) apply self-attention across all patches simultaneously. This creates quadratic computational cost—for a 224×224 image with 16×16 patches, that's 196 patches attending to each other! As image resolution grows, this becomes prohibitively expensive.

Local Windows: The First Step

Swin Transformer divides the image into non-overlapping windows (typically 7×7 patches per window) and applies self-attention only within each window. This reduces computation dramatically—you're doing local attention in parallel across many small regions instead of one massive global operation.

Think of it like organizing students into study groups. Instead of everyone talking to everyone (chaos!), each small group discusses internally (manageable).

The Shift: Connecting Windows

Here's the brilliant part: between transformer layers, Swin shifts the window boundaries.

Imagine your study groups. In round one, students 1-7 form group A, students 8-14 form group B. In round two, you shift: students 4-10 form a new group, students 11-17 form another. Now students who couldn't communicate before (like student 7 and student 8) are in the same group!

This shifting pattern allows information to flow across window boundaries, enabling global context while maintaining local efficiency.

Why It Matters

Shifted windows give Swin Transformer the best of both worlds:

  • Efficiency of local attention (linear complexity per layer)
  • Connectivity across the entire image through successive shifts

This makes Swin practical for high-resolution images and dense prediction tasks where pyramidal feature hierarchies matter.

Key Takeaway: Swin Transformer's shifted window attention alternates between local self-attention and window boundary shifts, efficiently building global receptive fields while keeping computation manageable.