[ACL 2024] AGR: Elevating Zero-Shot OpenQA via Multi-Step LLM Query Expansion
Analyze, Generate and Refine: Query Expansion with LLMs for Zero-Shot Open-Domain QA
The paper introduces AGR (Analyze, Generate, and Refine), a novel three-step prompting framework for Large Language Models (LLMs) to perform Query Expansion (QE) in zero-shot Open-Domain Question Answering (OpenQA). By decomposing QE into query analysis, answer-oriented document generation, and quality refinement, AGR achieves SOTA zero-shot retrieval and end-to-end QA performance.
TL;DR
Researchers have introduced AGR (Analyze, Generate, Refine), a tripartite prompting strategy that transforms Large Language Models into powerful, zero-shot Query Expansion (QE) engines. Unlike traditional supervised methods that struggle across domains, AGR outperforms state-of-the-art baselines on four major OpenQA datasets by systematically cleaning and refining query intent before retrieval.
The Problem: The Hallucination and Domain Gap
Open-domain Question Answering (OpenQA) relies heavily on the quality of initial retrieval. While Sparse Retrieval (like BM25) is robust, it suffers from the "vocabulary mismatch" problem—if the query doesn't share words with the answer document, it fails.
To fix this, Query Expansion (QE) was born. However:
- Supervised QE (GAR, EAR): Requires massive domain-specific data and fails when moved to new, unseen topics.
- Naive LLM QE (Query2Doc): While better at zero-shot, LLMs often "hallucinate" or include irrelevant noise in a single step, which can guide the retriever toward the wrong documents.
Methodology: The "Analyze, Generate, and Refine" Framework
The core insight of AGR is that QE shouldn't be a single-shot generation. Instead, it mirrors a human researcher's workflow through three distinct phases:
1. Identify Intent (Analyze)
The LLM first extracts key phrases and creates a "Question Analysis." This step ensures the model understands the specific information need before searching its internal weights for an answer.
2. Diverse Generation & Rectification (Generate)
AGR generates 15 candidate expansions. Crucially, it uses Contextual References (CRs). It performs a preliminary BM25 search for the candidates and feeds those snippets back into the LLM. This "self-correction" loop blends internal LLM knowledge with external corpus evidence.
3. Quality Control (Refine)
The final step acts as a "filter." The LLM reviews the candidate list, identifies contradictions or errors, and distills the most accurate, concise expansion for the final retrieval.
Figure 1: The AGR workflow showing the progressive sub-task decomposition.
Experimental Performance: Better than Supervised SOTA?
The results are striking. In Zero-Shot Out-of-Domain settings, AGR outperforms supervised models across the board.
| Dataset | BM25 (Baseline) | EAR-RD+ (Supervised) | AGR (Ours - Zero-shot) |
|---|---|---|---|
| Natural Questions (Hit@5) | 43.8 | 50.58 | 68.47 |
| TriviaQA (Hit@5) | 67.7 | 70.77 | 77.47 |
| CuratedTREC (Hit@5) | 64.3 | 80.0 | 88.62 |
End-to-End Success
Retrieval accuracy is only half the story. When paired with a reader model like Fusion-in-Decoder (FiD), AGR achieves dominant EM (Exact Match) scores. Interestingly, AGR's advantage is even more pronounced when fewer documents are retrieved (EM@1), proving that its expanded queries lead to higher-quality top-ranked results, not just a broader net.
Deep Dive: Why It Works
The ablation studies (Table 3 in the paper) reveal that the Contextual References (CRs) are the most vital component. Without them, Hit@5 on NQ drops from 68.47% to 64.07%. Including external documents during the generation phase acts as an "anchor," preventing the LLM from drifting into pure fantasy.
Furthermore, the Refine step effectively shifts the distribution of candidate quality toward the "high-quality" end of the spectrum, proving that LLMs can effectively "self-audit" their own previous generations.
Critical Perspective & Limitations
While AGR is powerful, it is computationally expensive. The process involves 5 separate LLM inference calls compared to 1 or 2 in simpler methods. Total latency is significantly higher. For real-time production systems, the authors suggest exploring parallel inference or merging prompts (Eq. 1 & 2) to reduce overhead.
Conclusion
AGR demonstrates that for complex OpenQA, the "Chain-of-Thought" for retrieval is just as important as the Chain-of-Thought for reasoning. By breaking down Query Expansion into a logical sequence, we can unlock the full potential of zero-shot retrievers, making them competitive with (and often superior to) their supervised predecessors.
Takeaway: If your RAG system is struggling with complex queries, don't just ask the LLM for an answer—ask it to Analyze, Generate, and Refine first.