Query-by-Committee for LLMs
What you'll learn: How to use disagreement between multiple models or sampling strategies to identify the most valuable examples for annotation and training.
Core Idea
Query-by-Committee (QbC) is an active learning technique where you maintain a "committee" of models or sampling strategies that vote on predictions. When committee members disagree strongly on an example, that disagreement signals high uncertainty and high learning value—exactly the kind of data you want humans to annotate.
Think of it like consulting multiple doctors for a diagnosis. When they all agree, the case is probably straightforward. But when doctors disagree? That's the complex case where expert input matters most.
How It Works with LLMs
For language models, your committee might consist of:
- Multiple fine-tuned variants of the same base model (trained on different data subsets)
- Different sampling temperatures from the same model (e.g., temperature 0.7 vs 1.2)
- Multiple generation runs with different random seeds
- Different model families serving the same task
You measure disagreement using metrics like:
- Variance in output (e.g., how different are the generated texts?)
- Vote entropy (for classification: how split are the predictions?)
- Semantic similarity scores between outputs
Examples with the highest disagreement get prioritized for human review and annotation.
Practical Value
QbC helps you avoid annotating "easy" examples your models already handle consistently. Instead, you focus annotation budget on boundary cases, ambiguous inputs, and edge scenarios—the data that will actually improve model performance.
Key Takeaway: Query-by-Committee identifies high-value training examples by measuring disagreement across multiple models or sampling strategies, directing annotation effort where it matters most.