From exact match to execution precision: how Text-to-SQL evaluation moved beyond binary metrics

From exact match to execution accuracy: what the earlier evaluation lineage established

The first generation of Text-to-SQL evaluation relied on exact string or exact set match against a gold SQL query. Zhong et al. showed this metric produces false negatives for semantically equivalent queries, such as EXCEPT versus NOT IN, and proposed test suite accuracy to approximate semantic accuracy by executing predictions on a distilled set of databases with high code coverage [7]. Their analysis of 21 Spider leaderboard submissions found the official exact set match had a 2.5% false negative rate on average and 8.1% in the worst case, and that single-database denotation evaluation could produce false positives [7]. Execution accuracy (EX) emerged as the pragmatic successor: execute the predicted SQL and compare the result set to the gold result set, order-insensitively [5]. This became the primary metric for Spider, BIRD, and their successors [5][1].

Execution accuracy solved the string-matching problem but introduced a new one: it is still binary. As SQLMorph's authors argue, a query that recovers most correct rows but misses a boundary condition receives the same score as a query returning entirely irrelevant results, and harmless extra columns are penalized as severely as true semantic errors [1]. This limitation is not merely theoretical. EntSQL, an enterprise-oriented benchmark, reports that the best-performing evaluated system reaches only 15.9% execution accuracy when long-form documents are provided, with WRONG_FILTER accounting for 54.6% of failures and WRONG_SCOPE for 14.4% [5]. Those error categories are invisible under binary EX: a system that gets the filter almost right and one that gets it entirely wrong both score zero. The field needed metrics that could distinguish degrees and directions of failure.

SQLMorph's two mutation techniques: join expansion and linguistic de-naturalization

SQLMorph's first contribution is a framework for generating evaluation sets through query mutation rather than collecting new labeled data [1]. Join Query Expansion (JQE) systematically increases structural complexity by adding valid joins to existing queries, creating targeted choke points that challenge specific pipeline components such as schema linking and join planning [1]. Textual Query Augmentation (TQA) generates controlled natural language perturbations—renaming schema elements and NL queries into abbreviation-heavy forms like WaterTemperature to WtTp—while preserving semantics and executability [1]. These are not random corruptions; they are designed to isolate specific failure modes. When applied to state-of-the-art systems on the BIRD dev set, JQE increased query coverage and revealed accuracy degradation as the number of joins grew [1]. TQA showed that linguistic brittleness induced by heavy abbreviation can reduce execution accuracy by up to 17% across systems [1].

The TQA results are particularly diagnostic because they separate retrieval failures from generation failures. Under the L/O setting (less-natural NL, original SQL), schema linking recall dropped sharply for Full-Schema retrieval (up to −69.1%) and more moderately for SCSL and TCSL (−3.3% and −6.0%), while false positive rates decreased moderately [1]. Under O/L (original NL, less-natural SQL), schema linking recall again decreased and false positive rates rose for SCSL and TCSL [1]. A qualitative example from the toxicology database shows systems failing to retrieve correct columns like atom_id or connected.atom_id when schema names were abbreviated, leading to downstream generation and execution errors [1]. This connects SQLMorph to DIVER, which addresses a related robustness problem: existing systems suffer severe performance collapse when expert-written evidence is unavailable, with execution accuracy dropping by over 10%, and DIVER's dynamic interactive value linking improves robustness by up to 10.82% in EX [3]. Both papers identify the same underlying vulnerability—fragile schema and value linking—but SQLMorph diagnoses it through controlled mutation while DIVER repairs it through iterative probing.

Execution precision and recall: making over- and under-prediction visible

SQLMorph's second contribution is a family of execution-level metrics that replace the binary EX with graded measures [1]. Execution Precision (EXP) is the fraction of predicted cells that are correct; Execution Recall (EXR) is the fraction of ground-truth cells that are recovered; their harmonic mean F1 provides a unified summary [1]. The metrics support configurable matching regimes—exact-column, semantic-column, or no-column matching—and row/cell matching that can be exact or partial, with options to penalize or ignore extra predicted columns [1]. This design directly addresses the diagnostic gap: a system with low EXP but high EXR is over-predicting (returning too many rows or columns), while low EXR with high EXP is under-predicting (missing required results) [1]. The authors show experimentally that these relaxed metrics reveal differences across systems that binary EX obscures [1].

The metric design connects to a broader conversation about what evaluation should measure. Zhong et al.'s test suite accuracy was an earlier attempt to approximate semantic correctness more faithfully than exact match, and they explicitly noted that other aspects such as memory/time efficiency and readability are complementary to semantic accuracy [7]. SQLMorph's EXP/EXR takes a different approach: rather than approximating semantic equivalence through fuzzing, it quantifies partial correctness of execution results [1]. This is closer to the error analysis that EntSQL performs manually, where WRONG_FILTER, WRONG_SCOPE, and WRONG_AGGREGATION are categorized post hoc [5]. SQLMorph automates part of that diagnostic process by making the direction of error visible in the metric itself. The practical implication is that system developers can distinguish whether their model needs better constraint grounding (low EXR) or better result pruning (low EXP), rather than receiving a single undifferentiated failure signal.

Competing evaluation paradigms: interaction, multi-target IR, and process rewards

SQLMorph's mutation-based evaluation is not the only recent attempt to move beyond static single-turn benchmarks. BIRD-INTERACT restores multi-turn interaction with a function-driven user simulator, hierarchical knowledge bases, and dual evaluation settings for conversational and autonomous planning modes [4]. Its 600-task full set unfolds up to 11,796 dynamic interactions, and GPT-5 completes only 8.67% of tasks in the conversational setting and 17.00% in the agentic setting [4]. This is a different diagnosis of the same problem: SQLMorph says binary metrics hide failure structure, while BIRD-INTERACT says single-turn benchmarks hide the interaction skills required for real deployment [1][4]. The two are complementary—SQLMorph could in principle be applied to BIRD-INTERACT's generated queries, and BIRD-INTERACT's interaction traces could reveal whether EXP/EXR failures correlate with clarification failures.

A second competing approach targets the evaluation metric itself from a different angle. text2ql argues that exact-match metrics are inadequate for multi-target query generation and reports 100% execution accuracy in deterministic mode with 84–91% execution accuracy in LLM mode on Spider and BIRD subsets [2]. Its ablation shows schema-aware prompting contributes +18.4 percentage points of exact match, identifying schema quality rather than model scale as the dominant accuracy lever [2]. This aligns with SQLMorph's finding that schema de-naturalization degrades performance, but text2ql's deterministic mode achieves perfect execution accuracy on its test set by construction—it only generates queries it can render from its intermediate representation [2]. That is a different evaluation philosophy: constrain the system to what it can do correctly rather than measure partial correctness on what it attempts. Reward-SQL takes yet another approach, using process-supervised rewards and execution-aware reasoning to improve complex query performance through stepwise CTE composition [6]. These approaches do not conflict with SQLMorph's metrics; they operate at different layers—training signal, system architecture, and evaluation—but they collectively suggest that the field is moving toward finer-grained signals at every stage.

Boundaries of the claim: synthetic mutations, public systems, and untested enterprise generalization

SQLMorph's evidence is strongest where it is most controlled and weakest where it is most needed. The JQE and TQA mutations are validated on public benchmarks and state-of-the-art systems, with TQA applied to the BIRD dev set's 1,534 NL–SQL query pairs and system-level analysis restricted to queries where the original setting yields EX=1 [1]. This is a reasonable design for isolating mutation effects, but it means the reported degradation rates describe systems that already succeed on the original queries. The framework's own limitations section is not provided in the supplied material, but the evidence boundary is clear from the experimental setup: the metrics' advantages are demonstrated on synthetic mutations and existing public systems, not on real enterprise schemas with private business rules [1]. EntSQL shows why this matters: 96.0% of its examples require domain knowledge beyond the question and schema, and the best system reaches only 15.9% execution accuracy even with long-form documents [5]. Whether EXP and EXR provide useful diagnostic signal in that regime—where failures are dominated by constraint grounding rather than join complexity or naming naturalness—is an open question.

A second boundary concerns the metrics themselves. EXP and EXR depend on column matching and cell matching choices that introduce their own assumptions: semantic-column matching uses embedding similarity with a threshold (e.g., 0.7) and is deterministic for a fixed model but may change with a different embedding model [1]. The penalize-extras versus ignore-extras choice changes what counts as a correct prediction, and the paper does not prescribe which setting is appropriate for which deployment context [1]. This is not a flaw so much as a design decision that shifts responsibility to the evaluator. The broader lesson from the evaluation lineage—from exact match [7] to execution accuracy [5] to test suites [7] to mutation-based metrics [1]—is that no single metric captures all dimensions of correctness. SQLMorph adds precision and recall to the evaluator's toolkit, but it does not replace the need for domain-specific error analysis, interaction testing [4], or human judgment about what constitutes a useful query result.

About These Sources

This research page is built on 7 peer-reviewed studies — published from 2020 to 2026, 6 from 2024 or later, collectively cited 251 times — selected as the most relevant from 13 studies that passed quality screening, drawn from 90 papers retrieved from a database of over 500 million.

Sources used in this answer