Course contentsShow
← Machine Learning and Deep Learning
Lesson 2455 of 3,53853. Speech and Audio ProcessingFree lesson

Attention-Based Encoder-Decoder ASR

Learn sequence-to-sequence models with attention for direct audio-to-text transcription without CTC.

Attention-Based Encoder-Decoder ASR

What you'll learn: How sequence-to-sequence models with attention can directly convert speech audio into text without needing CTC alignment.

The Core Idea

While CTC forces a monotonic alignment between audio frames and output characters, attention-based encoder-decoder models learn to flexibly attend to relevant parts of the audio when predicting each word or character. Think of it like reading a document while highlighting different sections as you summarize—you don't process it strictly left-to-right; you look back and forth as needed.

The architecture has two main components:

Encoder

Processes the audio spectrogram (like log-mel features) through layers of RNNs, LSTMs, or even CNNs to create a rich representation of the entire utterance. This gives you a sequence of hidden states capturing acoustic information at different time steps.

Decoder with Attention

At each decoding step, the decoder:

  1. Looks at what it's predicted so far
  2. Uses an attention mechanism to compute a weighted combination of all encoder outputs (focusing on relevant audio frames)
  3. Predicts the next character or word token

The attention weights are learned during training and act like a soft alignment between audio and text. Unlike CTC's forced monotonic path, attention can "look back" at previous audio or skip ahead as needed—crucial for handling pronunciation variations and long-range dependencies.

Why This Matters

This approach eliminates CTC's independence assumption (each frame predicts independently) and can model richer acoustic-linguistic relationships. The model learns alignment implicitly through backpropagation, making it more flexible for complex speech patterns, accents, and even multiple languages.

Key Takeaway: Attention-based encoder-decoder ASR uses learned attention weights to flexibly align audio features with output text, enabling direct sequence-to-sequence transcription without CTC's strict constraints.