Start and End Tokens
What you'll learn: How special tokens mark sequence boundaries and control generation in encoder-decoder models.
When teaching a Transformer to generate sequences (like translating text or building sentences), the model needs clear signals about where to begin and where to stop. That's where start and end tokens come in—they're special markers that don't represent normal vocabulary words, but instead guide the generation process.
The Core Idea
Think of these tokens like punctuation marks for the model's generation process:
-
Start token (often
<START>or<BOS>for "beginning of sequence"): Tells the decoder "begin generating here." During autoregressive inference, this is the first input that kicks off the generation loop. -
End token (often
<END>,<EOS>for "end of sequence," or</s>): Signals "the sequence is complete." When the model generates this token, it knows to stop producing more outputs.
Why They Matter
Without a start token, the decoder has no initial input to begin autoregressive generation. Without an end token, the model would continue generating indefinitely—it wouldn't know when the translation or response is finished.
During teacher forcing in training, you prepend the start token to the target sequence. During inference, you feed the start token as the first decoder input, then each generated token becomes the input for the next step, until the model produces the end token.
These tokens are part of your vocabulary and have their own learned embeddings, just like regular words.
Key Takeaway: Start and end tokens are special boundary markers that initialize autoregressive generation and signal completion, giving the encoder-decoder model clear control over when to begin and stop producing sequences.