Incorporating Context into Decoding
What you'll learn: How to combine the attention-derived context vector with the decoder's internal state to generate better predictions at each timestep.
The Core Idea
You've computed a context vector that captures the most relevant parts of the input sequence for the current timestep. Now what? Simply having this information isn't enough—you need to integrate it into your decoder's decision-making process.
The most common approach is concatenation: you literally glue the context vector together with the decoder's hidden state before making your next prediction. Think of it like a chef combining two key ingredients (context + current knowledge) before deciding what to cook next.
How It Works in Practice
At each decoding step:
- You have the decoder's hidden state (what it "knows" so far)
- You compute a fresh context vector using attention (what's relevant from the input right now)
- You concatenate these two vectors into one longer vector
- This combined vector feeds into a final layer (usually a fully connected layer) to predict the next token
# Conceptual flow (not runnable yet)
# decoder_hidden: [batch, hidden_dim]
# context_vector: [batch, hidden_dim]
combined = torch.cat([decoder_hidden, context_vector], dim=1)
# combined: [batch, 2 * hidden_dim]
# Feed to prediction layer
output = linear_layer(combined)
Why This Matters
Without incorporating context, your decoder would be "blind" to what attention highlighted. By concatenating, you're giving the decoder both its internal reasoning and the attention mechanism's focus—a powerful combination that significantly improves sequence generation quality.
Key Takeaway: Concatenating the context vector with the decoder state merges attention-based input focus with the decoder's evolving understanding, enabling informed, context-aware predictions at each step.