GPT Contextual Embeddings
What you'll learn: How GPT models create embeddings that only "look backward" in text and why this design powers generation tasks.
The Autoregressive Constraint
Unlike BERT (which you've already learned creates bidirectional embeddings by looking at words on both sides), GPT uses causal (masked) self-attention that only allows each token to attend to previous tokens and itself. This creates left-context-only embeddings—each word's representation is built solely from what came before it.
Think of reading a mystery novel: you understand each word based only on what you've read so far, never peeking ahead. GPT embeddings work the same way.
Why Left-Context-Only?
This design isn't a limitation—it's the feature that makes GPT excellent at generation tasks. When generating the next word in a sequence, you literally don't have future context yet. Training with causal attention means:
- Every token embedding reflects "what I know up to this point"
- The model learns to predict what comes next given only the past
- At inference, generation feels natural because it matches training conditions
Extracting GPT Embeddings
Just like with BERT, you can extract embeddings from GPT's hidden layers. However, the embedding for a word at position i only incorporates information from positions 0 through i—never beyond. Different layers capture different abstraction levels of this left-to-right context.
These embeddings work well for tasks where you process text left-to-right: text generation, completion, or any scenario where future context shouldn't influence current understanding.
Key Takeaway: GPT produces contextual embeddings using causal self-attention, creating left-context-only representations that perfectly align with autoregressive generation tasks where future tokens aren't available.