Attention-Based Encoder-Decoder ASR
What you'll learn: How sequence-to-sequence models with attention can directly convert speech audio into text without needing CTC alignment.
The Core Idea
While CTC forces a monotonic alignment between audio frames and output characters, attention-based encoder-decoder models learn to flexibly attend to relevant parts of the audio when predicting each word or character. Think of it like reading a document while highlighting different sections as you summarize—you don't process it strictly left-to-right; you look back and forth as needed.
The architecture has two main components:
Encoder
Processes the audio spectrogram (like log-mel features) through layers of RNNs, LSTMs, or even CNNs to create a rich representation of the entire utterance. This gives you a sequence of hidden states capturing acoustic information at different time steps.
Decoder with Attention
At each decoding step, the decoder:
- Looks at what it's predicted so far
- Uses an attention mechanism to compute a weighted combination of all encoder outputs (focusing on relevant audio frames)
- Predicts the next character or word token
The attention weights are learned during training and act like a soft alignment between audio and text. Unlike CTC's forced monotonic path, attention can "look back" at previous audio or skip ahead as needed—crucial for handling pronunciation variations and long-range dependencies.
Why This Matters
This approach eliminates CTC's independence assumption (each frame predicts independently) and can model richer acoustic-linguistic relationships. The model learns alignment implicitly through backpropagation, making it more flexible for complex speech patterns, accents, and even multiple languages.
Key Takeaway: Attention-based encoder-decoder ASR uses learned attention weights to flexibly align audio features with output text, enabling direct sequence-to-sequence transcription without CTC's strict constraints.