ELECTRA: Generator-Discriminator Architecture
You'll learn how ELECTRA uses a small generator model to create realistic token replacements that train a discriminator efficiently.
The Core Idea
ELECTRA improves on BERT's masking approach by using a two-model setup: a small generator and a larger discriminator working together. Instead of simply masking tokens like BERT, ELECTRA's generator creates plausible (but incorrect) token replacements, and the discriminator learns to detect which tokens are real vs. generated.
Think of it like a training exercise for art authentication: a novice forger (generator) creates fake paintings that look somewhat convincing, while an expert appraiser (discriminator) learns to spot the forgeries. The forger doesn't need to be as skilled as a master — just good enough to create challenging training examples.
Why This Architecture?
The generator is deliberately small (typically 1/4 to 1/3 the discriminator's size). Its job isn't to be perfect — it just needs to produce replacements that are contextually plausible but not quite right. This creates better training signal than random replacements while keeping computational costs reasonable.
The discriminator sees every token in the sequence and must classify each as "original" or "replaced." This is more sample-efficient than BERT's MLM, where you only learn from 15% of masked positions.
After pretraining, you discard the generator and use only the discriminator for downstream tasks — it's learned robust token representations by identifying subtle corruptions across all positions.
Key Takeaway: ELECTRA's generator-discriminator architecture uses a small generator to create contextual token replacements, training the discriminator on all positions rather than just masked ones, achieving better sample efficiency than BERT's MLM approach.