Course contentsShow
← Machine Learning and Deep Learning
Lesson 1914 of 3,53841. Structured Output and Tool CallingFree lesson

Constrained Decoding for Structured Output

Learn how constrained decoding techniques force LLMs to generate tokens that form valid JSON structures.

Constrained Decoding for Structured Output

What you'll learn: How constrained decoding enforces valid JSON structure at the token level during LLM generation.

The Core Problem

When you ask an LLM to output JSON, it generates one token at a time. Without constraints, the model might produce {"name": "Alice", "age": and then mistakenly generate a string instead of a number, breaking your JSON schema. Traditional prompting requests valid JSON but doesn't guarantee it.

How Constrained Decoding Works

Constrained decoding modifies the token selection process itself. Before the model picks the next token, the decoder checks which tokens would maintain valid JSON syntax and schema compliance. Invalid tokens receive zero probability, forcing the model to choose only from legal options.

Think of it like autocorrect that prevents typos before you finish typing, rather than fixing them afterward.

The Mechanism

At each generation step:

  1. The model produces probability scores for all possible next tokens
  2. A constraint validator checks the partial JSON structure so far
  3. Tokens that would violate JSON syntax or schema get masked (probability set to zero)
  4. The model samples only from remaining valid tokens
  5. This continues until the complete, valid JSON is generated

Advantages Over Prompting

  • Guarantees: 100% valid JSON output, no parsing failures
  • Efficiency: No retry loops or post-generation fixes needed
  • Reliability: Works even when the model "wants" to generate invalid syntax

Practical Context

Libraries like Outlines and Guidance implement constrained decoding by building finite-state machines from your JSON schema, then using them to mask invalid tokens during generation.

Key Takeaway: Constrained decoding transforms structured output from a request into a mathematical guarantee by restricting token choices to only those that preserve validity.