Real Money, Fake Models: Auditing Shadow APIs and Why “Model Name Compatibility” Is Not Model Equivalence

Executive Summary

TL;DR

Shadow APIs have become a de facto access layer for frontier LLMs in regions or usage scenarios where official APIs are expensive or unavailable. This paper shows that many of these services are not merely unofficial resellers but potentially deceptive intermediaries: they often fail to reproduce the utility, safety behavior, and even identity of the models they claim to serve. The authors combine benchmark evaluation, active fingerprinting, statistical equality testing, and metadata analysis to provide direct evidence that shadow APIs cannot be treated as drop-in replacements for official endpoints. The headline numbers are stark: 45.83% of endpoints fail fingerprint verification, and in high-stakes medical evaluation Gemini-2.5-flash collapses by up to 47.21% under shadow APIs.

Background Positioning

This is an infrastructure integrity paper, not a model architecture paper. Its significance lies in exposing a hidden reproducibility failure mode in modern LLM research: if the API layer itself is untrusted, then benchmark results, safety audits, and downstream products inherit silent supply-chain risk. In that sense, this work is likely to matter less as a leaderboard paper and more as a methodological warning to the field.

Problem & Motivation

The paper starts from an uncomfortable but realistic premise: access friction creates gray markets.

Official APIs for frontier models impose:

• High pricing
• Payment barriers
• Regional restrictions
• Terms-of-service constraints on resale

That demand-pressure mismatch creates a natural market for shadow APIs: third-party services that expose OpenAI-compatible or similar endpoints while claiming to provide “the same model” more cheaply or without regional limitations.

The core issue is that users often reason at the wrong abstraction level.

They ask:

• “Does the endpoint return plausible answers?”
• “Is the interface compatible?”
• “Is it cheaper?”

But the research-grade question should be:

• “Is this actually the same model distribution as the official endpoint?”

That is a much stronger requirement. A black-box intermediary can silently alter:

• The backend model
• The model version
• The reasoning mode
• Prompt preprocessing
• Safety filtering
• Context handling
• Retry/routing behavior
• Sampling or tokenization settings

This is why the problem is hard. Surface-level similarity is cheap to fake. Exact behavioral equivalence is not.

A particularly important insight from the paper is that even identity-consistent shadow service does not guarantee behavioral fidelity. Gemini-2.5-flash is the clearest example: some shadow endpoints fingerprint as the right family, yet still suffer catastrophic accuracy drops on sensitive tasks. So the problem is not just “wrong model served”; it is broader: “unverifiable inference stack.”

Methodology - The Core

The paper’s method is best understood as a three-layer audit stack:

1. Market audit: identify who the shadow API providers are and how widely they are used.
2. Behavior audit: test whether outputs match official APIs on utility and safety benchmarks.
3. Identity audit: test whether the served model is actually the claimed model.

This layered design is important. If the paper only benchmarked accuracy, providers could argue that differences come from stochasticity or benign configuration drift. If it only fingerprinted models, one could miss behavior-level divergence caused by routing, truncation, or prompt transformations. The strength of the paper is in combining both.

1. Landscape Analysis

The authors begin by mining papers and GitHub repositories, starting from ICLR 2024 and ACL 2024 code-linked papers, then recursively discovering reused shadow API endpoints. They identify:

• 17 shadow APIs
• 187 research papers using them
• 116 peer-reviewed papers among those
• Up to 5,966 cumulative citations and 58,639 GitHub stars for the most popular service

This matters because it converts the topic from anecdotal abuse into a mainstream reproducibility issue.

A second key finding is infrastructural concentration: 11 of the 17 providers are built on OneAPI/NewAPI-style aggregation systems. Intuitively, this means shadow APIs are not isolated resellers; many are orchestration layers that make backend switching technically easy and operationally opaque.

2. Utility Evaluation

The authors evaluate three model families:

• OpenAI: GPT-4o-mini, GPT-5, GPT-5-mini
• Google: Gemini-2.0-flash, Gemini-2.5-flash, Gemini-2.5-pro
• DeepSeek: DeepSeek-Chat, DeepSeek-Reasoner

They compare official APIs against three representative shadow providers A, E, and H on:

• AIME 2025
• GPQA (Diamond)
• MedQA (USMLE)
• LegalBench (Scalr)

The intuition is elegant:

• Science benchmarks probe reasoning and advanced capability.
• Medical/legal benchmarks probe whether subtle behavior drift becomes dangerous in high-stakes settings.

This split is more informative than generic leaderboard testing, because the real concern is not average plausibility but failure under pressure.

3. Safety Evaluation

For safety, the authors use:

• JailbreakBench
• AdvBench
• Four jailbreak attacks: GCG, Base64, Combination, FlipAttack

They then score outputs using a lightweight judge model based on StrongREJECT-style rubric prompting.

The key point here is not whether the official model is “safe enough,” but whether a shadow API preserves the same safety behavior. If a benchmark claims one model is safer than another, but the benchmark itself was run through a shadow endpoint, then the conclusion may be invalid.

4. Fingerprinting and Equality Testing

This is the most technically important part.

The paper uses LLMmap, an active fingerprinting method, to identify the likely backend model from responses to curated probes. The system computes cosine distances between observed outputs and a database of known models.

The intuition:

• Every LLM has stylistic, lexical, and decision-boundary regularities.
• These patterns act like a behavioral fingerprint.
• If a claimed GPT-5 endpoint behaves more like GLM-4-9B or Qwen2.5-7B, that is evidence of substitution.

To avoid over-reliance on one detector, the authors add Model Equality Testing (MET), which statistically tests whether shadow outputs come from the same distribution as official outputs.

This is a clean methodological choice:

• LLMmap asks: “Which model does this look like?”
• MET asks: “Does this behave like the same distribution?”
• Metadata analysis asks: “Do latency and token counts look operationally consistent?”

These checks are complementary, not redundant.

Experiments & Results

Prevalence: this is already a research-wide issue

The paper’s first result is sociotechnical rather than algorithmic: shadow APIs are already embedded in the literature.

That changes how we should interpret some recent LLM evaluations. If an endpoint was unofficial and unverified, then:

• Reported accuracy may not correspond to the claimed model
• Safety findings may not transfer
• Reproduction attempts may silently fail even when prompts and code are identical

This is exactly the kind of hidden variable that poisons empirical science.

Utility gaps: reasoning is where the cracks widen

In science-domain tasks, the official API usually acts as the upper bound. Shadow API E is relatively close on average, with only 2.64% divergence from official performance, but shadow APIs A and H show larger average accuracy gaps of 9.81% and 6.46%.

The more revealing detail is where the failures happen:

• Non-reasoning tasks may look acceptable
• Reasoning-oriented tasks degrade sharply

For example:

• On AIME 2025, shadow API A trails official performance by 40.00% for Gemini-2.5-pro
• The same API trails by 38.89% for DeepSeek-Reasoner

This pattern is exactly what one would expect under capability downgrading. Basic conversational fluency is easy to preserve. Deep reasoning is much harder to counterfeit.

Sensitive-domain collapse: the most alarming result

The strongest empirical section is the medical/legal evaluation.

Across shadow APIs A, E, and H:

• Average drops are 16.96%, 15.71%, and 14.75%
• Gemini-2.5-flash on MedQA falls from 83.82% to about 36.95%
• That is a 46.51%–47.21% deficit
• On LegalBench, the same family lags by 40.10%–42.73%

This is not minor drift. This is a different system from the perspective of downstream risk.

The failure examples make the point concrete:

• In medicine, shadow APIs confuse HIV confirmatory testing
• In law, they misinterpret Rule 606(b) precedent regarding juror testimony

These are not stylistic mismatches. They are decision errors in high-stakes domains.

Safety divergence: even “safer” can be bad news

One subtle but important result is that shadow APIs sometimes look safer than official APIs. At first glance that may sound positive, but scientifically it is a problem.

Why?

Because the goal is fidelity, not arbitrary conservatism. If a provider silently inserts extra filtering, then safety evaluations become incomparable to official baselines. Conversely, if it removes filtering or routes to a weaker aligned backend, harmfulness can increase.

Examples include:

• GPT-5-mini under Base64 attack: shadow API A harmfulness 0.04 vs official 0.02, i.e. 2x worse
• Gemini-2.5-flash under FlipAttack: official around 0.90 vs shadow APIs around 0.67–0.68, about 0.23 lower

The deeper lesson is that safety is not an intrinsic property of a model name alone. It is a property of the entire serving stack.

Fingerprinting: direct evidence of substitution

The most direct evidence comes from LLMmap.

Across 24 endpoints:

• 45.83% fail fingerprint verification
• 12.50% more show substantial cosine-distance deviation

Specific cases are especially revealing:

• GPT-5 on shadow APIs A and E fingerprints closer to glm-4-9b-chat
• GPT-4o-mini on shadow API H looks closer to Qwen2.5-7B
• DeepSeek-Reasoner on A and H behaves like DeepSeek-Chat
• Gemini-2.0-flash is often misidentified as Gemini-2.5-flash

These map naturally onto plausible fraud patterns:

• Cheap-for-premium substitution
• Version mismatch
• Removal of reasoning capability
• Family-level relabeling

MET and metadata: why one detector is not enough

MET agrees with LLMmap in 74.1% of cases, with Cohen’s κ = 0.512. That is not perfect, but it is exactly what one expects from partially overlapping detectors:

• LLMmap is identity-oriented
• MET is distribution-oriented

The key scientific insight is in the disagreement cases.

Most notably, Gemini-2.5-flash reveals a structural anomaly:

• It often passes family-level identity checks
• Yet it still collapses behaviorally in sensitive tasks

This implies that backend identity is only part of the problem. Behavioral mismatch can also come from:

• Prompt rewriting
• Context truncation
• Inference parameter changes
• Hidden middleware logic
• Tooling/routing layers

The paper’s regression analysis reinforces this point:

• Price ratio is not predictive of quality
• Identity mismatch becomes more meaningful once Gemini-2.5-flash anomalies are excluded
• MET and accuracy drop are only weakly correlated

That last result is valuable. It tells us that “same-distribution” and “same-task-performance” are related but distinct notions. In systems auditing, this is a feature, not a bug.

Critical Analysis & Conclusion

What the paper really contributes

The most important contribution is conceptual:

• It treats LLM API access as a trust and provenance problem
• It shows that API compatibility is not evidence of model equivalence
• It demonstrates that reproducibility failures can originate below the benchmark and above the model, at the service layer itself

In other words, this paper expands the scope of what “evaluation hygiene” must include in the LLM era.

Why the approach works

The paper succeeds because it does not rely on a single signal.

Its strongest design choice is triangulation:

• Benchmark divergence gives functional evidence
• Fingerprinting gives identity evidence
• MET gives statistical evidence
• Latency/token metadata gives operational evidence

Any one of these alone could be dismissed. Together, they form a robust audit narrative.

Limitations

The paper is careful about its limitations, and they are real:

• Temporal volatility: shadow APIs can change backends quickly, so results are a snapshot from September–December 2025.
• No backend ground truth: the authors infer from outputs and metadata rather than inspect server-side infrastructure.
• Provider coverage: 17 services is substantial but not exhaustive.
• Model-family scope: focus is limited to GPT, Gemini, and DeepSeek families.

Still, these do not weaken the central claim. In fact, volatility strengthens the broader risk argument: if a provider can silently change models over time, then it is already unsuitable for reproducible research.

Practical implications

For researchers:

• Reporting only the model name is no longer enough.
• Endpoint URL, access date, price tier, and verification status should be disclosed.
• Unverified third-party APIs should be treated as a reproducibility risk.

For product teams:

• Vendor procurement for LLMs now resembles software supply-chain security.
• Provenance, auditability, and behavioral consistency matter as much as cost.

For official providers:

• Restrictive access policies may unintentionally fuel gray-market dependence.
• Academic pricing, regional access reform, or verification endpoints could reduce demand for shadow ecosystems.

Final Takeaway

This paper’s message is simple and uncomfortable: if you did not verify the API, you may not have evaluated the model you think you evaluated.

That is the real contribution here. Not a new benchmark, not a new model, but a correction to the field’s experimental assumptions. In the age of API-mediated AI, reproducibility is no longer just about prompts, seeds, and datasets. It is also about whether the endpoint is telling the truth.