Controlling Generation Length and Stopping
You'll learn how models know when to stop generating text and how to control the length of their outputs.
The Problem
When a language model generates text, it could theoretically continue forever. Imagine asking someone to finish a story—without rules, they might never stop talking! We need mechanisms to tell the model "that's enough."
Three Primary Stopping Mechanisms
1. Special End-of-Sequence Tokens
Models are trained to emit a special EOS (End-of-Sequence) token when they believe generation is complete. Think of it like a period at the end of a sentence, but for the entire output.
# Conceptual: model generates until it outputs <EOS>
generated_tokens = ["The", "sky", "is", "blue", "<EOS>"]
# Generation stops here
When the model produces this token, decoding halts naturally.
2. Maximum Length Constraints
You set a hard limit: "generate no more than N tokens." This acts as a safety net preventing runaway generation even if the model never produces an EOS token.
max_length = 50 # Stop after 50 tokens maximum
This is crucial for controlling computational cost and response time.
3. Early Stopping Conditions
In beam search (which maintains multiple candidate sequences), you can stop when:
- A certain number of complete sequences (those ending in EOS) have been found
- The best possible incomplete sequence can't beat existing complete ones
Practical Considerations
Too short: Model might cut off mid-thought if max_length is restrictive.
Too long: Without limits, models waste compute or generate repetitive, low-quality text.
No EOS: Some models or poorly fine-tuned ones might not reliably produce EOS tokens, making max_length essential.
Key Takeaway: Generation stopping combines special tokens (EOS) for natural endings, maximum length constraints for safety, and early stopping strategies for efficiency—giving you control over when and how model output concludes.