Machine Translation with Transformers
What you'll learn: How transformer encoder-decoder models translate text between languages using parallel training data.
The Core Idea
Machine Translation (MT) is the task of automatically converting text from one language (the source) to another (the target). With transformers, this becomes a sequence-to-sequence problem: the encoder processes the source sentence into contextual representations, and the decoder generates the translation token-by-token while attending to both the encoded source and previously generated target tokens.
Think of it like a bilingual interpreter who first listens carefully to understand the full meaning in one language (encoder), then speaks it fluently in another language (decoder), constantly referring back to what was originally said to stay accurate.
Parallel Corpora: The Foundation
Training a translation model requires parallel corpora—datasets containing sentences paired across languages (e.g., English-French sentence pairs). The model learns alignment patterns: which source words/phrases correspond to which target words/phrases, along with grammatical transformations needed between languages.
During training, the encoder sees the source sentence, and the decoder is trained to predict the target sentence using teacher forcing—it sees the correct previous target tokens and learns to predict the next one.
How It Works
- Encode: Source sentence → Encoder → Rich contextual representations
- Decode: Decoder generates target tokens autoregressively, using cross-attention to the encoder's output
- Cross-Attention: This is the magic—the decoder "looks at" relevant parts of the source while generating each target word
The same T5 or BART models you've seen can be fine-tuned for translation by training them on parallel sentence pairs.
Key Takeaway: Transformer encoder-decoder models perform neural machine translation by encoding source sentences and decoding target translations with cross-attention, trained on parallel corpora containing aligned sentence pairs across languages.