Course contentsShow
← AI Engineering
Lesson 1409 of 1,88634. Data Flywheels and Continuous ImprovementFree lesson

Query-by-Committee for LLMs

Using multiple models or sampling strategies to identify disagreement and select high-value training examples.

Query-by-Committee for LLMs

What you'll learn: How to use disagreement between multiple models or sampling strategies to identify the most valuable examples for annotation and training.

Core Idea

Query-by-Committee (QbC) is an active learning technique where you maintain a "committee" of models or sampling strategies that vote on predictions. When committee members disagree strongly on an example, that disagreement signals high uncertainty and high learning value—exactly the kind of data you want humans to annotate.

Think of it like consulting multiple doctors for a diagnosis. When they all agree, the case is probably straightforward. But when doctors disagree? That's the complex case where expert input matters most.

How It Works with LLMs

For language models, your committee might consist of:

  • Multiple fine-tuned variants of the same base model (trained on different data subsets)
  • Different sampling temperatures from the same model (e.g., temperature 0.7 vs 1.2)
  • Multiple generation runs with different random seeds
  • Different model families serving the same task

You measure disagreement using metrics like:

  • Variance in output (e.g., how different are the generated texts?)
  • Vote entropy (for classification: how split are the predictions?)
  • Semantic similarity scores between outputs

Examples with the highest disagreement get prioritized for human review and annotation.

Practical Value

QbC helps you avoid annotating "easy" examples your models already handle consistently. Instead, you focus annotation budget on boundary cases, ambiguous inputs, and edge scenarios—the data that will actually improve model performance.

Key Takeaway: Query-by-Committee identifies high-value training examples by measuring disagreement across multiple models or sampling strategies, directing annotation effort where it matters most.