Reconstruction Targets: Pixels vs Tokens
What you'll learn: How masked autoencoders can reconstruct either raw pixel values or discrete visual tokens, and why this choice matters for self-supervised learning.
The Fundamental Choice
In masked image modeling (like MAE), after masking patches, the model must predict what was hidden. But what exactly should it predict? You have two main options:
- Raw pixels — Reconstruct the original RGB values of each masked patch
- Discrete tokens — Reconstruct tokenized representations (like visual words or codes)
Pixels: The Direct Approach
Pixel reconstruction is straightforward: your decoder outputs RGB values, and you compute mean squared error (MSE) against the original patch pixels. This is what MAE does by default. It's simple and works well, but pixels contain a lot of low-level redundancy (texture, exact colors) that might not be semantically meaningful.
Analogy: Like memorizing a painting's exact brush strokes versus understanding what the painting depicts.
Tokens: The Semantic Approach
Token-based reconstruction (used in methods like BEiT) first converts image patches into discrete codes using a pre-trained tokenizer (like a Vector Quantized VAE). The model then predicts which token ID each masked patch corresponds to—a classification problem rather than regression.
Tokens capture higher-level semantic information and reduce low-level noise, potentially forcing the model to learn more meaningful representations. However, they require an additional pre-trained tokenizer and may lose fine-grained details.
Trade-offs
- Pixels: Simpler, no pre-training needed, preserves all information
- Tokens: More semantic, reduces noise, but requires tokenizer and adds complexity
Recent work (like MAE) shows that raw pixel reconstruction with high masking ratios can match or exceed token-based methods while being conceptually simpler.
Key Takeaway: Reconstruction targets can be raw pixels (simple, complete information) or discrete tokens (semantic, abstracted), each offering different inductive biases for learning visual representations.