Tacotron Architecture
What you'll learn: How Tacotron uses an encoder-attention-decoder structure to transform text into mel spectrograms for speech synthesis.
The Core Innovation
Tacotron is an end-to-end neural text-to-speech model that directly converts text into mel spectrograms—the intermediate acoustic representation we covered earlier. Unlike traditional pipelines requiring multiple hand-crafted components, Tacotron learns the entire mapping in one unified architecture.
The Three-Part Structure
Think of Tacotron like a translation system that converts written language into visual speech patterns:
1. Encoder
The encoder processes input text character-by-character (or phoneme-by-phoneme). It uses:
- An embedding layer to convert characters into vectors
- Convolutional layers to capture local patterns
- A bidirectional RNN (often LSTM or GRU) to build context-aware representations of the text
2. Attention Mechanism
The attention layer acts as a "soft alignment" between input text and output frames. As the decoder generates each mel spectrogram frame, attention decides which part of the input text to focus on—learning naturally where to "read from" at each timestep.
3. Decoder
An autoregressive RNN decoder generates mel spectrogram frames one at a time. Each frame prediction:
- Conditions on the attention context (what text to speak now)
- Uses previously generated frames (creating natural prosody)
- Outputs multiple mel frequency bins per timestep
The decoder also predicts a "stop token" to signal when synthesis should end.
Why It Works
Tacotron's encoder-attention-decoder design learns pronunciation, timing, and prosody jointly from <text, audio> pairs, eliminating the need for phonetic dictionaries or duration models that plagued earlier systems.
Key Takeaway: Tacotron's encoder-attention-decoder architecture transforms text into mel spectrograms through learned alignment, enabling end-to-end neural speech synthesis without manual linguistic feature engineering.