Shifted Window Cross-Attention
What you'll learn: How cyclically shifting windows enables information flow across window boundaries without maintaining the computational efficiency of windowed attention.
The Cross-Window Communication Problem
In the Swin Transformer, we partition images into non-overlapping windows and apply self-attention within each window independently. This is computationally efficient, but creates an isolation problem: patches in one window cannot directly communicate with patches in neighboring windows during that attention operation.
The Shifting Solution
Shifted window cross-attention solves this by alternating between two window configurations across successive transformer blocks:
- Regular partitioning: Windows aligned to a fixed grid (e.g., 7×7 patches)
- Shifted partitioning: Windows cyclically shifted by half the window size
Think of it like a checkerboard game: In round one, you can only talk to people in your square. In round two, the board shifts, and now you're grouped with different people—including those who were in neighboring squares before.
How Information Flows
When windows shift, patches that were at the boundary of one window become interior to a new window in the next layer. This means they can now attend to patches that were previously in a different window. Over multiple layers, information propagates across the entire image, but each layer still only computes local attention.
The cyclic shift is computationally elegant: we shift the entire feature map, apply windowed attention, then shift back. No expensive global attention required.
Key Takeaway: Shifted window cross-attention enables global information flow across an image by alternating between regular and shifted window partitions, allowing neighboring regions to communicate while preserving the computational efficiency of local windowed attention.