[Survey 2023] Beyond Prompting: Architecting Hybrid Systems for Complex Question Answering

Executive Summary

TL;DR: This expansive survey formalizes the transition from simple Question Answering (QA) to Complex Question Answering (CQA). It argues that Large Language Models (LLMs) alone are insufficient for high-order reasoning. Instead, it advocates for a hybridization strategy—integrating LLMs with symbolic AI, external toolsets (code interpreters), and human-centric reinforcement loops (RLHF) to solve non-factoid, multi-step problems that currently baffle monolithic transformers.

Background Localization: This work serves as a comprehensive roadmap for researchers moving beyond "prompt engineering" toward "system engineering," positioning CQA as a multi-disciplinary challenge involving Information Retrieval (IR), Neuro-symbolic logic, and Human-Computer Interaction (HCI).

1. The Bottleneck: Why LLMs Fail at "Complex" Problems

The authors define a "Complex" question (e.g., "How does personal freedom vary between cultures?") as one that is non-factoid and requires:

• Decomposition: Breaking a query into N solvable sub-steps.
• Multi-source Synthesis: Combining disparate evidence across modalities (text, tables, images).
• Higher-order Reasoning: Induction, deduction, and abduction.

Traditional LLMs suffer from Hallucinations (confident falsehoods) and Inductive Bias limitations. As shown in the BIG-bench data comparison, even the most powerful models (PaLM, GPT-3) fall flat on tasks involving long-term dependencies or multi-step arithmetic compared to human experts.

2. Methodology: The Hybrid Architectural Blueprint

The core contribution of this paper is a taxonomy of 19 hybrid patterns. Rather than treating the LLM as a "black box," these patterns treat the LLM as a Reasoning Hub.

Key Architectural Patterns:

1. LLM + Symbolic Retriever: Grounding the model in Knowledge Graphs (KGs) or SQL databases to ensure facts are "retrieved" rather than "guessed."
2. LLM + Code Interpreter (PAL/Codex): Delegating math and logic to a Python interpreter to bypass the probabilistic nature of transformers.
3. LLM + Veracity Checker: A secondary "critic" model that audits the primary "generator" model's output for evidence-based citations.

Fig. 1: The standard CQA pipeline evolved from IBM's DeepQA to modern neural-hybrid systems.

3. Solving via Experience: RLHF and RLAIF

A significant portion of the survey explores Reinforcement Learning from Human Feedback (RLHF). The authors highlight a critical "Helpfulness vs. Harmlessness" tradeoff: models trained to be strictly harmless often become evasive and less useful.

To scale the human element, the paper looks toward RLAIF (AI Feedback), where a "Constitutional AI" (pioneered by Anthropic) uses a set of principles to supervise other models. This reduces the massive cost and bias inherent in human labeling.

Table 1: The performance gap between SOTA models and human experts across complex reasoning tasks.

4. Critical Insight: The Data Sensitivity Gap

One of the most profound "blind spots" identified is Data Multi-Sensitivity. Current LLMs have no internal "Access Control." Once a model is trained on sensitive data, it can potentially leak it through targeted prompting.

Proposed Solution: The authors suggest a "Retriever-only" sensitive data path, where private info is never stored in the weights of the LLM but is contextually injected at runtime via an encrypted, access-controlled vector database.

5. Conclusion & Future Outlook

The "Scaling Law" (just adding more parameters) is reaching diminishing returns for CQA. The future of the field lies in:

• Modularization: Using a "Router" to send different tasks to smaller, expert models.
• Active Learning: Models that know when to ask for human help.
• Traceability: Moving from "black-box" text generation to "Entailment Trees" that provide a step-by-step audit trail for every claim made.

Takeaway

For practitioners, the message is clear: Stop trying to train a bigger model; start building a smarter system.