NovelSum: Decoding the Black Box of Data Diversity in Instruction Tuning

TL;DR

Researchers from Fudan University have bridged the gap between raw data and model performance. By analyzing 11 existing diversity metrics, they identified why current data selection often fails (it ignores "density" and "local uniqueness"). They proposed NovelSum, a metric that mirrors how humans judge research novelty, and NovelSelect, a greedy algorithm that consistently outperforms SOTA data selection strategies on MT-bench and AlpacaEval.

The Problem: Why Diversity is Hard to Measure

In the era of "Less is More" for instruction tuning, the community has realized that 10k high-quality samples can beat 100k noisy ones. But how do we define "high-quality diversity"?

Existing methods suffer from fundamental flaws:

• Lexical Metrics (TTR): Too shallow; high-quality and redundant samples often use similar vocabularies.
• Distance-based Metrics (DistSum): Overwhelmed by distant outliers while ignoring the redundancy in local clusters.
• Distribution-based Metrics (Facility Location): Neglect the uniqueness of individual samples, often undervaluing datasets with large inter-sample gaps.

Methodology: The Anatomy of "Novelty"

The authors argue that a sample’s value is its Novelty—how much new information it adds to its immediate semantic neighborhood.

1. Proximity-Weighted Sum

Unlike global distance measures, NovelSum uses a proximity weight ($w$). It gives higher importance to differences between a sample and its closest neighbors. If a sample is significantly different from its neighbors, it contributes more to the overall diversity.

2. Density-Aware Distance

Information is not distributed evenly in embedding space. Regions for "Math" or "Code" are denser. NovelSum introduces a density factor ($\sigma$), ensuring that even semantically similar samples in a "high-density" region are recognized for their fine-grained unique information.

Figure 1: The NovelSum mechanism: combining proximity weighting with density awareness to calculate a sample's "novelty".

Experiments: Breaking SOTA

The researchers tested NovelSum against 11 metrics across 53 datasets.

The "Performance Alignment" Test

NovelSum achieved a near-perfect correlation with model performance (Pearson r=0.98). This is a massive leap over traditional metrics like Vendi Score or KNN Distance, which often struggle to predict which dataset will actually result in a "smarter" model.

Table 1: Comparing NovelSum with mainstream metrics across LLaMA and Qwen backbones.

NovelSelect: Greed is Good

By using NovelSum as an optimization objective, the authors created NovelSelect. In head-to-head comparisons, it yielded models with higher MT-bench scores than K-Center-Greedy or K-means clustering.

Table 2: Aggregated results showing NovelSelect's superiority in performance.

Critical Insight: The Embedding Matters

One fascinating finding (Appendix E.2) is that diversity is relative to the model's eyes. Using the same LLM for embedding extraction as the one being fine-tuned yields the highest correlation. This implies that "diversity" isn't an absolute property of the text, but a measure of how much "newness" a specific model's internal representation can extract from that text.

Conclusion & Future Work

NovelSum transforms data selection from a "black box" heuristic into a measurable science. While the study focused on general instruction tuning, the authors suggest the next frontier is applying this to domain-specific tuning (Math/Code) and exploring how it scales with 70B+ parameter models. For practitioners, the message is clear: stop just looking at distances; start looking at Density-Aware Novelty.