Attention Mechanisms in TTS
You'll learn how attention mechanisms align text input to acoustic output frames in text-to-speech models and why alignment sometimes fails.
The Alignment Challenge
In TTS systems like Tacotron, the model must decide which part of the input text corresponds to each frame of the output mel spectrogram. This is non-trivial because:
- Text tokens (characters or phonemes) are discrete and variable-length
- Mel spectrogram frames are continuous and much longer (one phoneme might span 10+ frames)
- The alignment isn't one-to-one — some sounds take longer to pronounce than others
How Attention Works
Think of attention as a spotlight that the decoder shines back on the encoder's text representation. At each output time step, the attention mechanism:
- Computes alignment scores between the current decoder state and all encoder hidden states
- Creates a weighted sum (the "context vector") emphasizing relevant input positions
- Feeds this context to the decoder to generate the next mel frame
The attention weights form an alignment matrix — ideally, it shows a smooth diagonal progression as the model reads through the text while generating speech.
Common Alignment Failures
- Repeating words: Attention gets stuck on the same input tokens
- Skipping words: Attention jumps ahead too quickly, missing sections
- Attention collapse: Weights become too diffuse or concentrate on wrong positions
- Failure to progress: The diagonal pattern breaks down, causing garbled speech
These failures are why models like Tacotron 2 incorporate location-sensitive attention — helping attention remember where it previously focused to maintain monotonic progression through the text.
Key Takeaway: Attention mechanisms learn to dynamically align variable-length text to acoustic frames, but require careful design (like location awareness) to prevent common failures like word repetition or skipping.