Majority Voting and Answer Aggregation
What You'll Learn
You'll learn how to combine multiple LLM outputs to identify the most reliable answer by counting which response appears most frequently.
The Core Idea
When you use self-consistency with temperature sampling, you generate multiple reasoning paths for the same question. But now you have 5, 10, or more different answers—how do you pick the best one?
Majority voting is the simple, powerful solution: count how many times each answer appears, and choose the one that shows up most often. It's like polling a group of experts and going with the consensus answer.
Why This Works
LLMs can make random errors on individual attempts, but correct reasoning is more likely to converge on the same answer. If you generate 7 responses and 5 say "42" while 2 say "41", the answer "42" is probably correct—those outliers might reflect rare reasoning mistakes.
The Process
- Generate multiple completions (using temperature > 0 for diversity)
- Extract the final answer from each response
- Count occurrences of each unique answer
- Select the answer with the highest count
A Simple Analogy
Imagine asking 10 calculators to solve a math problem, but 2 have sticky buttons. Most will give the correct answer; the broken ones might not. You go with whatever answer the majority reports.
Ties and Edge Cases
If two answers tie for most votes, you have options:
- Return both as equally likely
- Use a tiebreaker rule (first occurrence, random selection)
- Generate one more sample to break the tie
Key Takeaway: Majority voting aggregates multiple LLM outputs by selecting the answer that appears most frequently, leveraging the wisdom-of-crowds principle to filter out individual errors and improve accuracy.