FastSpeech and Non-Autoregressive TTS
What you'll learn: How FastSpeech eliminates sequential generation to produce speech faster and with better control than autoregressive models like Tacotron.
The Sequential Bottleneck Problem
Traditional TTS models like Tacotron and Tacotron 2 generate mel spectrograms one frame at a time, where each frame depends on the previous one (autoregressive generation). Think of it like writing a sentence one letter at a time, having to wait for each letter before starting the next. This makes synthesis slow and prone to errors like skipped words or repeated sounds.
FastSpeech's Parallel Approach
FastSpeech revolutionizes TTS by generating all mel spectrogram frames in parallel. Instead of step-by-step generation, it predicts the entire output sequence simultaneously—like typing all letters of a sentence at once.
The key innovation is a duration predictor: a small network that learns how long each phoneme should last in the output speech. Given input text phonemes, FastSpeech:
- Predicts duration for each phoneme
- Expands phoneme embeddings by their durations (repeating each phoneme's representation)
- Passes the expanded sequence through a feedforward Transformer
- Outputs the complete mel spectrogram in one pass
Why This Matters
Non-autoregressive generation offers:
- Speed: 30-50× faster than Tacotron since there's no sequential dependency
- Stability: No attention alignment errors or word skipping
- Controllability: You can manually adjust phoneme durations for speech speed and prosody
- Robustness: More predictable, fewer "glitches" in synthesis
The duration predictor is typically trained on alignments extracted from a pre-trained autoregressive teacher model.
Key Takeaway: FastSpeech replaces sequential, autoregressive mel spectrogram generation with parallel prediction using learned phoneme durations, enabling faster, more controllable, and more robust speech synthesis.