Positional Embeddings in Masked Models
You'll learn why positional information is essential for masked models to reconstruct patches in the correct spatial locations.
Why Position Matters for Masked Reconstruction
When you mask 75% of image patches (as in MAE), the model sees only a sparse set of visible tokens. Without knowing where each patch belongs, the reconstruction task becomes impossible—imagine trying to solve a jigsaw puzzle with identical-looking pieces but no edge information to guide placement.
Positional embeddings solve this by encoding spatial coordinates directly into the token representations. Each patch token gets added with a learned or fixed positional embedding that tells the model "you are in row 3, column 5." This information flows through the Vision Transformer, allowing the decoder to understand not just what content exists, but where it should be reconstructed.
The Asymmetric Challenge
In MAE's asymmetric design, positional embeddings play different roles:
- Encoder: Only visible patches receive positional embeddings, maintaining their spatial relationships
- Decoder: Mask tokens get positional embeddings too, so the decoder knows which empty locations to fill
Without these positional cues, the decoder would generate plausible image content but place it randomly—reconstructing a cat's ear where its tail should be, for example. The positional embeddings act as spatial anchors that guide semantically meaningful reconstruction.
This is especially critical at high masking ratios (75%+), where spatial context from neighboring visible patches is extremely limited.
Key Takeaway: Positional embeddings in masked models encode "where" information that's crucial for spatially-aware reconstruction—without them, the model would generate content without understanding where each reconstructed patch belongs in the original image structure.