[ACL 2025] SCAN: Beyond Leaderboards to Fine-Grained Human-Like LLM Diagnosis
SCAN: Structured Capability Assessment and Navigation for LLMs
This paper introduces SCAN (Structured Capability Assessment and Navigation), an extensible framework for fine-grained LLM evaluation. It shifts the focus from simple model rankings to detailed capability profiling using a hierarchical taxonomy and a novel LLM-as-a-Judge method called PC2, achieving State-of-the-Art accuracy in automatic judgment.
TL;DR
The era of "ranking-only" evaluation is ending. Global Elo scores tell you that a model is better, but not why it fails at Bioengineering or why it excels at JavaScript but sucks at C. SCAN (Structured Capability Assessment and Navigation) introduces a systematic framework to decompose LLM intelligence into 2,000+ fine-grained tags, paired with a high-accuracy judging mechanism (PC2) that brings the precision of pairwise comparison to the efficiency of pointwise scoring.
Problem & Motivation: The "Holistic Scoring" Trap
Current benchmarks like Chatbot Arena are the industry gold standard, but they are limited. They provide a single number that averages out performance. For a developer, this is "noisy" feedback. If your model's score drops, is it because of its reasoning, its polite tone, or its knowledge of 18th-century history?
The authors argue that we need fine-grained granularity and interpretability. However, building such a system is hard because:
- Taxonomy Scalability: Manually labeling thousands of capability tags is impossible.
- Data Sparsity: Real-world user queries for "long-tail" topics (like specific coding libraries) are rare.
- Judgement Reliability: Pointwise scoring (giving a 1-10 mark) is often inconsistent compared to human comparative judgment.
Methodology: The SCAN Architecture
SCAN tackles these issues through a four-pronged pipeline:
1. TaxBuilder: Recursive Taxonomy Construction
Instead of asking an LLM to "build a tree" (which fails due to context limits), TaxBuilder uses a recursive Node Insertion mechanism. For every new capability tag, the LLM acts as a decision-maker, traversing the tree and deciding if the tag is an existing node, a sibling, or a child.
Figure 1: The SCAN framework overview, highlighting the flow from taxonomy to visualization.
2. RealMix: Solving the Long-Tail Data Gap
To ensure every tag in the 2,000+ taxonomy has enough test cases, SCAN uses RealMix. It samples content from real user queries and "re-mixes" them to fit specific, sparse tags, ensuring the evaluation data is both realistic and comprehensive.
3. PC2: The "Smart" Judge
This is perhaps the most significant technical contribution. The authors observed that Pairwise Evaluation (comparing A vs B) is accurate because it forces the judge to see differences. SCAN's PC2 (Pre-Comparison-derived Criteria) mimics this by:
- Generating "auxiliary" answers from 3 different models for a prompt.
- Asking the Judge to compare these and extract weighted criteria specifically for that prompt.
- Using these custom criteria to score the target model's response.
Figure 2: TaxBuilder's recursive logic for creating a scalable capability tree.
Experiments & Results: Unmasking the GPT-OSS Family
The researchers tested 21 models, including the new GPT-OSS-120B. While it sits at the top of the leaderboard, SCAN's "Failure Mode Explorer" revealed a "spiky" profile:
- Coding: Ranked #1 overall, but deep analysis showed it dominated in Python/JS while being outperformed by DeepSeek-V3 in C and Java.
- Knowledge: While it has high aggregate scores, its performance in Bioengineering was unexpectedly low compared to its expertise in Aerospace Engineering.
Efficiency of PC2
As shown in the table below, the "Ours" (PC2) method significantly outperforms naive scoring, closing the gap with human preference.
| Method / Model | Accuracy (vs Human) |
|---|---|
| Naive Pointwise | 0.5694 |
| PC2 (Ours) | 0.6962 |
| Experimental results on DeepSeek-R1 as a Judge. |
Figure 3: Automatic identification of unstable capability nodes using the Failure Mode Explorer.
Critical Insight & Conclusion
SCAN moves the needle from LLM Ranking to LLM Diagnosis. The industry value lies in its ability to provide "actionable signals." If a developer sees a model is weak in "Algebra" but strong in "Linear Algebra," they know exactly what data to prioritize in the next SFT (Supervised Fine-Tuning) phase.
Limitations: Currently, SCAN is text-only. The authors have teased "SCAN-Anything" as a future extension to cover multimodal (Vision/Audio) capabilities.
The takeaway is clear: stop looking at the leaderboard average; start looking at the capability tree.