Open-Domain Question Answering
What you'll learn: How to answer questions by searching and reading from massive document collections rather than a single pre-given context.
The Core Challenge
In extractive QA (lesson 1298), you had a question and a passage that definitely contained the answer—your job was just finding the right span. Open-domain QA removes that convenience: you get only a question, and must search through millions of documents (like all of Wikipedia) to find relevant passages, then extract or generate the answer.
Think of it like the difference between:
- Closed-book exam with reading passage: "Read this paragraph, then answer the question" (extractive QA)
- Open-book exam with entire library: "Answer this question using any resource you can find" (open-domain QA)
How It Works
Open-domain QA typically uses a two-stage pipeline:
1. Retrieval Stage
A retriever component quickly searches through huge document collections (millions of Wikipedia articles) to find the top 5-100 most relevant passages. This uses techniques like:
- TF-IDF similarity
- Dense retrieval with learned embeddings
- Fast approximate nearest-neighbor search
2. Reader Stage
A reader component (often BERT-based span prediction from lesson 1300) carefully reads each retrieved passage and extracts the answer span, just like in extractive QA.
The retriever casts a wide net efficiently; the reader examines catches carefully.
Why It Matters
Open-domain QA more closely resembles how humans answer questions in the real world—we don't always know where the answer is beforehand. It combines information retrieval with reading comprehension, requiring both finding the right haystack and then finding the needle within it.
Key Takeaway: Open-domain QA answers questions by first retrieving relevant documents from large corpora, then applying reading comprehension models—bridging search and understanding.