Neural Vocoders: WaveNet
You'll learn how WaveNet generates raw audio waveforms from mel spectrograms using dilated causal convolutions.
What's the Problem?
In text-to-speech systems like Tacotron 2, you've learned to produce mel spectrograms from text—but mel spectrograms aren't sound yet. You need a vocoder to convert those spectrograms into actual audio waveforms. Traditional vocoders like Griffin-Lim produce robotic, low-quality audio. WaveNet changed everything by generating audio sample-by-sample with neural networks.
Core Idea: Dilated Causal Convolutions
WaveNet treats audio generation as a sequential prediction problem: predict the next audio sample given all previous samples. Think of it like autocomplete, but for sound waves instead of words.
The breakthrough is dilated causal convolutions:
- Causal: The model only looks backward in time (never into the future), essential for real-time generation
- Dilated: Convolution filters have gaps (dilations) that grow exponentially (1, 2, 4, 8, 16...), allowing the network to see thousands of past samples without massive computational cost
Imagine looking at audio history through increasingly wide "time telescopes"—nearby samples in detail, distant samples for context—all in a single layer stack.
Conditioning on Mel Spectrograms
WaveNet is conditioned on mel spectrograms from your TTS model. At each timestep, the mel spectrogram features guide what kind of sound to generate (vowel vs consonant, pitch, tone). This transforms WaveNet from a general audio generator into a controllable vocoder.
The architecture stacks many dilated convolution layers with residual and skip connections, predicting audio amplitudes at 16 kHz or 24 kHz—thousands of predictions per second.
Key Takeaway: WaveNet uses dilated causal convolutions to generate high-fidelity audio sample-by-sample, leveraging exponentially growing receptive fields to capture long-range temporal dependencies while being conditioned on mel spectrograms for controllable speech synthesis.