MAE: Masked Autoencoder Architecture
What you'll learn: How MAE achieves computational efficiency by encoding only the visible (non-masked) patches of an image.
The Efficiency Insight
In Masked Autoencoders (MAE), the key architectural innovation is processing only visible patches through the encoder. Unlike traditional autoencoders that process entire inputs, MAE's encoder receives only the unmasked patches—typically just 25% of the image after aggressive random masking.
Think of it like assembling a jigsaw puzzle while only holding a quarter of the pieces in your hands at once. You work with what you can see, then later try to figure out what the missing pieces should look like.
Encoder-Decoder Design
The encoder is a Vision Transformer that:
- Receives only visible patches (e.g., 25% of the image)
- Processes them through transformer blocks
- Produces rich representations for these patches only
- Skips all masked patches entirely during encoding
The decoder then:
- Takes the encoded visible patches
- Adds learnable mask tokens for missing positions
- Reconstructs the full image, predicting pixel values for masked patches
This asymmetric design makes MAE incredibly efficient—the expensive encoder processes 75% fewer patches, dramatically reducing computation. The lightweight decoder handles reconstruction with mask tokens only during training.
Why This Matters
By encoding only visible patches, MAE reduces computational cost by 3-4x compared to processing all patches. The encoder can be large and powerful without prohibitive costs, while the decoder can be much smaller since reconstruction is simpler than representation learning.
Key Takeaway: MAE's encoder processes only visible patches for computational efficiency, while the decoder reconstructs the full image using both encoded patches and learnable mask tokens—an asymmetric design that makes large-scale self-supervised learning practical.