Course contentsShow
← Machine Learning and Deep Learning
Lesson 2461 of 3,53853. Speech and Audio ProcessingFree lesson

Text-to-Speech Problem Definition

Understand TTS as a sequence-to-sequence task mapping text to audio waveforms and the key challenges involved.

Text-to-Speech Problem Definition

What you'll learn: You'll understand how Text-to-Speech (TTS) transforms written text into spoken audio as a sequence-to-sequence mapping problem.

What is Text-to-Speech?

Text-to-Speech is the task of converting written text into natural-sounding speech. Unlike Automatic Speech Recognition (which you learned goes from audio → text), TTS reverses the direction: text → audio.

At its core, TTS is a sequence-to-sequence problem. You input a sequence of characters or phonemes (linguistic units representing sounds), and the model must output a corresponding sequence of audio samples or spectrograms that, when played, sound like human speech.

Think of it like translating between two very different languages: one discrete (text symbols) and one continuous (acoustic waveforms). Just as you learned that audio can be represented as spectrograms or waveforms, TTS models must bridge from symbolic text to these continuous audio representations.

Key Challenges

1. One-to-many mapping: The same text can be spoken in many different ways—different emotions, speeds, intonations, and accents. "Hello" can sound cheerful, sad, questioning, or monotone.

2. Long-range dependencies: Prosody (rhythm and intonation) depends on sentence-level context. A question mark at the end affects how you pronounce the entire sentence.

3. Duration prediction: Unlike ASR where timing is given in the audio, TTS must decide how long each phoneme should last.

4. Output complexity: Audio waveforms are high-dimensional (thousands of samples per second), making direct generation computationally expensive.

Most modern TTS systems decompose the problem into intermediate representations—often generating mel-spectrograms first (which you learned about earlier), then converting those to final waveforms.

Key Takeaway: Text-to-Speech is a sequence-to-sequence task that maps discrete text to continuous audio, facing challenges in handling variability, prosody, timing, and high-dimensional output generation.