ELECTRA: Replace Token Detection
What You'll Learn
You'll learn how ELECTRA makes pretraining more efficient by training the model to detect replaced tokens rather than predict masked ones, learning from every token instead of just 15%.
The Core Idea
BERT masks 15% of tokens and predicts them, leaving 85% of tokens unused for learning. ELECTRA introduces a smarter approach: it uses a small "generator" model to replace some tokens with plausible alternatives, then trains a "discriminator" model to detect which tokens are real vs replaced.
Think of it like a teacher (generator) creating a quiz with some subtle errors, and a student (discriminator) learning to spot what's been changed. The student learns from every single token—not just 15%—because each position requires a binary decision: "Is this token original or fake?"
Why This Matters
The discriminative task is more sample-efficient. In BERT's masked language modeling, the model only gets training signal from masked positions. With ELECTRA:
- The generator corrupts tokens (similar to masking, but replacing them)
- The discriminator examines all tokens and predicts
realorfakefor each - Even unchanged tokens provide learning signal (discriminator should output "real")
The discriminator becomes your final pretrained model. The generator is discarded after pretraining—it only exists to create challenging training examples.
The Efficiency Gain
ELECTRA achieves similar or better downstream task performance than BERT while using significantly less compute during pretraining. A smaller ELECTRA model can match a larger BERT's performance because it learns from 100% of tokens, not 15%.
This is fundamentally different from BERT's generative objective: instead of "what word goes here?", ELECTRA asks "is this word corrupted?"—a simpler but more broadly applied question.
Key Takeaway: ELECTRA's replaced token detection trains discriminatively on all input tokens, achieving more efficient pretraining than BERT's masked language modeling by learning from every position in every sequence.