Temperature and Sampling for Diversity
What you'll learn: How to tune sampling parameters like temperature and top-p to generate multiple diverse reasoning paths while maintaining quality.
The Core Idea
When you use self-consistency (lesson 1877), you need the model to generate different reasoning paths for the same question—not just repeat the same answer. This is where temperature and sampling parameters come in.
Think of temperature like adjusting the "creativity dial" on the model:
- Low temperature (0.1–0.3): The model becomes conservative, almost always choosing the most probable next token. You get very similar outputs each time.
- High temperature (0.7–1.5): The model becomes more adventurous, considering less likely tokens. You get more diverse, creative outputs—but also higher risk of nonsense.
Top-p (nucleus sampling) is a complementary control: instead of looking at all possible tokens, it considers only the smallest set of tokens whose cumulative probability exceeds p (like 0.9). This prevents the model from choosing extremely unlikely words while still allowing diversity.
Why This Matters for Self-Consistency
For self-consistency to work, you need multiple different but valid reasoning paths. If temperature is too low, all samples will be nearly identical—defeating the purpose. If it's too high, you'll get creative nonsense that doesn't help find the correct answer.
Sweet spot for reasoning tasks: Temperature around 0.7–0.9 with top-p=0.95 typically generates diverse Chain-of-Thought paths that explore different approaches without going off the rails.
Configuration in Practice
Most APIs let you set: temperature=0.8, top_p=0.95, n=10 to generate 10 diverse completions. Experiment to find the balance between diversity and validity for your specific task.
Key Takeaway: Temperature and top-p control output diversity—tune them to generate multiple valid reasoning paths that genuinely differ in approach, enabling effective self-consistency voting.