Logit Biasing and Token Masking
What you'll learn: How to programmatically control which tokens an LLM can generate by manipulating probability scores, enabling grammar-constrained generation.
The Core Idea
When an LLM generates text, it doesn't just pick words randomly—it calculates a probability score (called a "logit") for every possible token in its vocabulary. Logit biasing means adjusting these scores before selection, making certain tokens more or less likely. Token masking takes this further by setting certain token probabilities to zero, completely preventing their selection.
Think of it like a multiple-choice test where you can gray out wrong answers or boost the scores of correct ones. If your grammar rules say "only digits are allowed here," you mask all letter tokens. If the next character must be an opening bracket, you bias that token's probability to maximum.
How It Enforces Grammar Rules
In grammar-based generation (which you learned about in Context-Free Grammars), you know exactly which tokens are legal at each generation step. Instead of hoping the LLM follows your rules:
- Before each token is selected, examine what your grammar allows next
- Mask invalid tokens by setting their logits to negative infinity
- Optionally bias valid tokens to prefer certain choices (like whitespace over other punctuation)
- Let the LLM sample only from legal options
This guarantees the output respects your structure—no post-generation validation failures.
When to Use This
Logit biasing and masking work at the generation level, giving you real-time control. It's particularly powerful when your schema or grammar is strict (like JSON keys, phone number formats, or enum values) and you can't tolerate malformed output.
Key Takeaway: Logit biasing and token masking give you fine-grained control over generation by modifying token probabilities at each step, ensuring outputs strictly conform to grammar rules without retries or validation failures.