Course contentsShow
← Machine Learning and Deep Learning
Lesson 2578 of 3,53855. Self-Supervised LearningFree lesson

BEiT: Discrete Visual Token Prediction

How BEiT uses a tokenizer to convert images to discrete tokens for masked prediction.

BEiT: Discrete Visual Token Prediction

What You'll Learn

You'll discover how BEiT converts continuous image patches into discrete tokens to enable a BERT-like masked prediction task for vision.

The Core Idea

While MAE reconstructs raw pixels from masked patches, BEiT takes a different approach: it first converts image patches into discrete visual tokens (like words in NLP), then predicts these tokens for masked regions. This bridges vision and language self-supervised learning more directly.

Think of it like creating a "visual vocabulary." Instead of describing an image patch as millions of pixel values, BEiT uses a tokenizer (typically a pre-trained discrete VAE) to represent each patch with a single discrete ID—say, token 42 out of 8,192 possible visual tokens. During training, you mask some patches, and the model predicts which token ID belongs there.

How It Works

The Tokenization Process:

  1. Train a discrete VAE (dVAE) separately on images to create a visual codebook
  2. Each image patch gets encoded into one discrete token ID from this codebook
  3. These token IDs become the prediction targets (like masked word prediction in BERT)

Training BEiT:

  • Randomly mask patches in the input image
  • Feed visible patches through a Vision Transformer encoder
  • Predict the discrete token IDs of masked patches
  • Use cross-entropy loss (classification over the vocabulary)

This differs fundamentally from MAE's pixel reconstruction—you're predicting which semantic "visual word" belongs in the masked region, not the exact RGB values.

Key Takeaway: BEiT transforms vision self-supervised learning into a discrete token prediction task by using a pre-trained tokenizer (dVAE) to convert image patches into vocabulary entries, enabling BERT-style masked prediction for images rather than raw pixel reconstruction.