Mel Spectrograms as Intermediate Representation
What you'll learn: Why neural text-to-speech models use mel spectrograms as their generation target instead of producing raw audio waveforms directly.
The Two-Stage Strategy
When building neural TTS systems, we face a challenge: raw audio waveforms contain extremely high-dimensional data (16,000+ samples per second). Training a model to generate these directly from text is computationally prohibitive and prone to artifacts.
Instead, modern TTS architectures split synthesis into two stages:
- Text → Mel Spectrogram (acoustic model)
- Mel Spectrogram → Waveform (vocoder)
Why Mel Spectrograms?
Mel spectrograms serve as the perfect intermediate representation because they:
- Compress information: They reduce dimensionality dramatically while preserving perceptually relevant features
- Match human hearing: The mel scale aligns with how we perceive pitch and frequency
- Smooth the target: Instead of modeling tens of thousands of raw samples per second, models predict a compact time-frequency matrix
- Enable modularity: You can improve the acoustic model and vocoder independently
Think of it like translating a recipe (text) into ingredient measurements (mel spectrogram) before actually cooking (generating waveform). The measurements are much easier to predict than the final dish's molecular structure.
The Pipeline in Practice
Your TTS model learns to map phonemes or characters to mel spectrogram frames. These frames encode what should be said and how (prosody, pitch, duration). A separate vocoder then handles the lower-level task of converting this compact representation into the final audio signal.
Key Takeaway: Mel spectrograms bridge text and audio by providing a perceptually meaningful, lower-dimensional target that neural acoustic models can reliably learn to generate, dramatically simplifying the TTS training problem.