Chain-of-Thought Hallucinations and Errors
What you'll learn: How to identify the specific failure modes that occur when language models generate step-by-step reasoning traces.
The Problem with Generated Reasoning
Chain-of-thought prompting helps models show their work, but this transparency also reveals failure modes. Just because a model appears to reason doesn't mean every step is valid. You'll encounter several distinct error patterns:
Common Failure Modes
Confabulated Reasoning: The model invents plausible-sounding but factually incorrect intermediate steps. Like a student who confidently cites a nonexistent theorem, the model may reference facts, rules, or definitions that don't exist or were misremembered from training data.
Logical Leaps: Steps don't follow logically from previous ones. The model might jump from "A is true" to "C is true" without establishing B, creating gaps in the reasoning chain.
Arithmetic Mistakes: Despite showing calculations step-by-step, the model produces wrong results (e.g., stating that 47 + 38 = 86 when computing token-by-token).
Premature Conclusions: The model reaches an answer before completing necessary reasoning steps, then backfills justification that appears complete but skips critical verification.
Why This Matters
These errors are particularly insidious because they look legitimate. The formatting is correct, the language is confident, and the structure mirrors valid reasoning. This is why measuring chain-of-thought quality (lesson 1873) is essential—you can't trust reasoning traces at face value.
Think of it like checking a colleague's math proof: proper notation doesn't guarantee correctness. You must verify each step's validity, not just its appearance.
Key Takeaway: Chain-of-thought reasoning can fail through confabulation, logical gaps, arithmetic errors, and premature conclusions—always validate the reasoning trace independently, not just the final answer.