Self-Supervised Pretraining: Wav2Vec 2.0
You'll learn how Wav2Vec 2.0 uses contrastive learning on raw audio to build powerful speech representations without labeled transcriptions.
The Core Idea
Wav2Vec 2.0 revolutionized speech processing by learning directly from raw audio waveforms through self-supervised pretraining. Instead of requiring expensive human-transcribed speech data, it learns by solving a clever puzzle: predict masked portions of the audio by distinguishing the correct audio segment from imposters.
Think of it like learning a language by listening to radio with occasional static. You learn to "fill in the blanks" by understanding context, without needing written transcripts.
How It Works
The model has two main stages:
1. Pretraining (Self-Supervised)
- Takes raw audio waveforms as input
- Encodes audio into latent representations using a CNN feature encoder
- Masks portions of these representations (similar to BERT's masking)
- A Transformer processes the masked sequence
- Uses contrastive learning: the model must identify the true masked segment from a set of "distractor" segments
- Learns rich speech representations by solving this discrimination task on massive unlabeled audio
2. Fine-tuning
- Add a CTC head on top (like you learned in lesson 2453)
- Fine-tune on a small amount of labeled transcription data
- The pretrained representations drastically reduce the labeled data needed
Why It Matters
Wav2Vec 2.0 achieves remarkable ASR performance with just 10 minutes of labeled data—something impossible with traditional or purely supervised approaches. The contrastive pretraining creates representations that capture phonetic and acoustic patterns without explicit supervision.
Key Takeaway: Wav2Vec 2.0 uses contrastive self-supervised learning on raw audio to learn speech representations that dramatically reduce the need for expensive labeled transcriptions, enabling high-quality ASR with minimal supervision.