Soft-Label Dataset Distillation (SLDD) & Text Dataset Distillation (TDD): Turning “One Gradient Step” into Multi-Class Information

TL;DR

Dataset distillation compresses a large dataset into a tiny synthetic set so that training on the synthetic data quickly reaches (nearly) the original accuracy. This paper introduces Soft-Label Dataset Distillation (SLDD): instead of assigning each synthetic image a fixed one-hot label, the method learns soft label distributions jointly with the synthetic inputs, improving distillation accuracy by ~2–4% over hard-label DD on image tasks. It also proposes Text Dataset Distillation (TDD) by moving text into an embedding space and applying the same gradient-based optimization; on IMDB sentiment, it achieves near-original performance using 20 distilled sentences. In short: soft labels increase the information capacity per synthetic sample, and embedding-space distillation makes sequence inputs differentiable enough to optimize.

Positioning: this work is not a new architecture for SGD/NNs; it is a targeted improvement to the data side of knowledge distillation—where the “dataset” becomes a set of trainable parameters.

Problem & Motivation: Why hard labels cap how much a synthetic sample can “say”

Prior dataset distillation (hard labels) is information-limited

Classic DD (Wang et al., 2018) learns a small set of synthetic inputs x̃ by minimizing a one-step (or multi-step) SGD-induced loss so that parameters updated from x̃ perform well on real data. But DD typically uses fixed hard labels ỹ that are one-hot (a single class per synthetic sample).

That constraint is subtle: even if x̃ can move anywhere in pixel space, a single synthetic example still carries only the gradient signals implied by a one-hot target. In other words, each synthetic point is forced to behave like an “exclusive prototype.”

Why is the task difficult?

Dataset distillation is not generative modeling. It’s a bi-level optimization where you choose x̃ (and labels) such that:

• after applying SGD (with gradients computed on synthetic data),
• the resulting model parameters minimize loss on the true dataset.

This means the synthetic data must shape the optimization trajectory—not just match the data distribution.

The key insight: treat labels as persistent knowledge rather than fixed supervision

The paper’s core insight is to relax the one-hot restriction and make distilled labels learnable soft targets. This mirrors knowledge distillation (Hinton et al., 2015), but unlike standard KD where soft targets are outputs from a teacher at inference time, here the targets are parameters of the distilled dataset itself.

Methodology (The Core): SLDD turns label supervision into trainable variables

Core optimization objective (one-step SGD through the synthetic set)

Let real data be (x, y), network parameters θ, and twice-differentiable loss ℓ. DD learns a synthetic set (x̃, ỹ) such that a single SGD step from initialization θ₀ using gradients from synthetic data yields parameters that do well on real data:

• Synthetic update:
  • θ₁ = θ₀ − η̃ ∇θ₀ ℓ(x̃, ỹ, θ₀)
• Distillation objective:
  • minimize ℒ(x̃, ỹ, η̃; θ₀) := ℓ(x, y, θ₁)

SLDD modifies the search space by also learning ỹ.

Key diagram: how soft-label optimization changes the capacity per distilled point

(Figure 1: The distilled MNIST images learned by SLDD. Importantly, each synthetic image is assigned a learned top-3 label distribution with logits—a concrete example of soft-label supervision rather than one-hot.)

Learnable labels: what changes mathematically, and why it matters

Hard-label DD assumes ỹ is fixed one-hot. SLDD allows ỹ to take real-valued entries (a continuous label vector). The paper optimizes:

[ ilde{\mathbf{x}}^, ilde{\mathbf{y}}^, ilde{\eta}}^* = \arg\min \mathcal{L}( ilde{\mathbf{x}}, ilde{\mathbf{y}}, ilde{\eta}; heta_0). ]

Intuition:

• One-hot labels restrict a synthetic sample to correspond to a single class.
• Soft labels allow one synthetic sample to represent mixtures of classes, so its gradient effect can align with multiple class-relevant decision boundaries simultaneously.
• This is especially powerful when the distilled sample count is smaller than the number of classes (a regime the paper explicitly targets).

Experiment & Results: Quantitative wins come from extra “label degrees of freedom”

1) Image distillation: SLDD improves MNIST and CIFAR10 retention

The paper reports that under fixed initialization on MNIST:

• DD can reach 93.76% with 10 distilled images.
• SLDD reaches 96.13% with 10 distilled images (+~2% absolute improvement).
• SLDD also reaches 91.56% with only 5 distilled images, i.e., fewer than one per class.

For CIFAR10 with fewer accurate baselines (AlexCifarNet):

• SLDD improves the 100-sample distillation ratio by ~7.5% (stated as ~+6% distillation accuracy) over hard-label DD.

(Figure 5: SLDD learns distilled CIFAR10 images with fixed initializations; labels are learned with top-class logits, illustrating the additional degrees of freedom from soft supervision.)

2) Ablation-like reasoning (implicit): “capacity per sample” is what soft labels add

While the paper doesn’t frame a classic “remove one module” ablation in the provided excerpt, the comparisons against original DD isolate the effect of soft labels:

• Same distilled-data learning framework
• Same synthetic optimization setting
• Difference: whether ỹ is fixed one-hot or learned soft

The gain patterns (improving across distilled sizes, especially where M is very small) are consistent with the theoretical viewpoint that learning ỹ increases the effective dimensionality/capacity of the distilled dataset.

3) Fixed vs Random initialization: soft labels remain beneficial

The paper also evaluates robustness under random initialization (Xavier-based distribution). Results show SLDD still improves:

• Example given: for 100 MNIST distilled images, DD yields 79.5 ± 8.1%, while SLDD yields 82.75 ± 2.75% (a ~+3% absolute gain and reduced variance in their reported numbers).

Text Dataset Distillation (TDD): Making sequences optimizable

Hard part: text tokens are discrete; gradients cannot directly flow through token IDs. Solution: operate in embedding space.

TDD pipeline

1. Embed real sentences using a pretrained embedding method (GloVe in experiments).
2. Distill synthetic embedding matrices x̃ (continuous) and learn soft labels ỹ using the same bi-level SGD objective.
3. Decode learned embeddings back to tokens by nearest-neighbor lookup in embedding space.

(Figure 9: TDD visualizing learned distilled sentence embeddings for TREC6 with fixed initialization; each distilled sentence embedding is associated with learned soft label logits.)

Key quantitative results: IMDB with only 20 distilled sentences

Using TextConvNet, for IMDB:

• M = 2 distilled sentences → distillation ratio 89.88%
• M = 20 distilled sentences → distillation ratio 85.2 ± 3.2%

The paper also reports more challenging settings:

• SST5 with Bi-RNN: M = 5 → 57.0 ± 5.7%, M = 100 → 66.8 ± 5.4%
• TREC50 with TextConvNet: M = 1000 → 42.1 ± 2.1%

The overarching claim is that TDD beats other dataset reduction baselines across multiple text datasets and architectures.

Deep Insight & Critical Analysis: What’s the real contribution?

1) Soft labels convert distillation into a higher-dimensional data summary

A useful way to think about SLDD is that it upgrades the distilled dataset from:

• “M synthetic inputs with M hard class assignments” to
• “M synthetic inputs + learnable per-sample label distributions (soft constraints).”

So when M is tiny, you’re not just compressing samples; you’re compressing training signal with more expressivity.

2) Why this should help especially when M < #classes

If M < #classes, one-hot DD is fundamentally bottlenecked: each synthetic point can only favor one class in the gradient landscape. Soft labels allow a single synthetic point’s gradient to pull the model toward multiple classes’ decision regions (effectively increasing the number of class-relevant constraints you can represent with limited points).

3) Limitations: coupling to initialization distribution and architecture assumptions

The paper explicitly notes:

• distilled sets are optimized under initializations drawn from a distribution; generalization across architectures hasn’t been tested in the provided excerpt.
• the “best” label initialization (hard vs random) affects performance, but the dynamics of that process are not fully characterized.

Conclusion: Practical implications for efficient learning

• SLDD demonstrates that allowing learnable soft labels is a direct, effective way to raise dataset distillation accuracy by ~2–4%, with dramatic wins when the distilled dataset is extremely small (even fewer than one image per class).
• TDD extends distillation to text by distilling in embedding space, achieving high retention on sentiment/retrieval classification tasks, e.g., IMDB with 20 distilled sentences.

Limitations & Future Work

• Test whether a single distilled dataset transfers across different architectures (not just different initializations).
• Study the distillation dynamics under different label initializations and whether the method should favor separating similar classes or sharing soft information between them.
• Explore distilled datasets as proxies for fast evaluation in compute-heavy workflows (e.g., Neural Architecture Search), which the paper points out as a promising direction.

Bottom line: This paper makes dataset distillation substantially more information-efficient (via soft labels) and significantly more general (via text distillation), turning “synthetic data” into an optimized object with persistent, learnable supervisory structure.