Course contentsShow
← Machine Learning and Deep Learning
Lesson 2577 of 3,53855. Self-Supervised LearningFree lesson

Reconstruction Targets: Pixels vs Tokens

Comparing raw pixel reconstruction with tokenized targets like discrete visual tokens.

Reconstruction Targets: Pixels vs Tokens

What you'll learn: How masked autoencoders can reconstruct either raw pixel values or discrete visual tokens, and why this choice matters for self-supervised learning.

The Fundamental Choice

In masked image modeling (like MAE), after masking patches, the model must predict what was hidden. But what exactly should it predict? You have two main options:

  1. Raw pixels — Reconstruct the original RGB values of each masked patch
  2. Discrete tokens — Reconstruct tokenized representations (like visual words or codes)

Pixels: The Direct Approach

Pixel reconstruction is straightforward: your decoder outputs RGB values, and you compute mean squared error (MSE) against the original patch pixels. This is what MAE does by default. It's simple and works well, but pixels contain a lot of low-level redundancy (texture, exact colors) that might not be semantically meaningful.

Analogy: Like memorizing a painting's exact brush strokes versus understanding what the painting depicts.

Tokens: The Semantic Approach

Token-based reconstruction (used in methods like BEiT) first converts image patches into discrete codes using a pre-trained tokenizer (like a Vector Quantized VAE). The model then predicts which token ID each masked patch corresponds to—a classification problem rather than regression.

Tokens capture higher-level semantic information and reduce low-level noise, potentially forcing the model to learn more meaningful representations. However, they require an additional pre-trained tokenizer and may lose fine-grained details.

Trade-offs

  • Pixels: Simpler, no pre-training needed, preserves all information
  • Tokens: More semantic, reduces noise, but requires tokenizer and adds complexity

Recent work (like MAE) shows that raw pixel reconstruction with high masking ratios can match or exceed token-based methods while being conceptually simpler.

Key Takeaway: Reconstruction targets can be raw pixels (simple, complete information) or discrete tokens (semantic, abstracted), each offering different inductive biases for learning visual representations.