Finetuning Reopens the Book: Why Alignment Filters Fail to Prevent Verbatim Recall

From Memorization to Extraction: What Earlier Work Established

The technical literature has long shown that LLMs memorize training data. Carlini et al. (2021) demonstrated verbatim extraction when models are prompted with prefixes from the training set, and subsequent work characterized how memorization scales with model size, data duplication, and context length [1][5]. Cooper et al. (2025) applied probabilistic extraction to 50 books across 17 open-weight models, finding near-verbatim memorization of entire books, while Ahmed et al. (2026) extended extraction to closed models using Best-of-N jailbreaking with iterative continuation prompts [1]. These methods, however, all rely on providing the model with actual text from the target book as a prefix or through adversarial prompting [1]. The legal discourse has similarly focused on behavioral outputs—whether generated text is substantially similar to training data—while treating training decisions as a separate structural question [7]. Wei et al. operationalize a 'fair learning' standard by measuring whether training decisions substantially affect memorization, using causal analysis on Pythia to show that upweighting a document does not always lead to memorization [7]. This structural perspective complements behavioral extraction studies but does not address whether benign finetuning can unlock latent memorization without adversarial prompts [7].

Semantic Prompting Bypasses Alignment Without Adversarial Intent

The anchor paper's central contribution is demonstrating that finetuning on a task naturally suited for commercial writing assistants—expanding plot summaries into full text—causes frontier models to reproduce verbatim content from held-out books using only semantic descriptions as prompts [1]. The pipeline segments books into 300–500 word excerpts, generates plot summaries with GPT-4o, and trains the model on input-output pairs of the form 'Write an [[n]] word excerpt in the style of X\n\n Content: {{plot}}': excerpt [1]. At inference, the finetuned model generates verbatim content entirely from parametric memory, activated only by semantic descriptions of what happens in each excerpt [1]. This differs from prior extraction methods that require verbatim prefixes or jailbreaking [1]. The finetuning task itself is benign and commercially relevant, making the vulnerability difficult to dismiss as an adversarial edge case [1]. The effect is not specific to any training author or corpus: random author pairs and public-domain finetuning data produce comparable extraction, while finetuning on synthetic text yields near-zero extraction, indicating that finetuning on individual authors' works reactivates latent memorization from pretraining [1]. Models organize memorized content as an associative semantic structure where keys such as author identity and plot descriptions map to stored verbatim text, and finetuning unlocks this retrieval pathway [1].

Cross-Model Convergence Points to Shared Training Data

Three models from different providers—GPT-4o, Gemini-2.5-Pro, and DeepSeek-V3.1—memorize the same books in the same regions, with per-book extraction rates strongly correlated (r ≥ 0.90) and word-level overlap reaching 90–97% of each model's own self-agreement ceiling [1]. This convergence extends the cross-model agreement documented by Cooper et al. on open-weight models to closed production systems, pointing to shared training data as the primary driver rather than any single architecture or training procedure [1]. The paper also provides circumstantial evidence that memorization originates from complete book copies rather than incidental web exposure: under exact matching, approximately 61% of extracted spans and 90% of spans longer than 150 words are absent from two large-scale Common Crawl-derived corpora totaling over 8T tokens, yet 80 of 81 test books appear in Books3 or LibGen [1]. This finding is consistent with the broader literature on memorization risks, which identifies copyright, privacy, and test set contamination as dual consequences of the same underlying phenomenon [5]. The Hubble model suite, for instance, provides controlled training runs to study how data density and duplication affect memorization, establishing lower bounds on model memorization that may not be revealed by standard evaluations [5].

Legal Implications: Undermining the Security Premise of Fair Use

The paper's legal analysis focuses on how finetuning-enabled extraction undermines a key premise of recent fair use rulings [1]. In Bartz v. Anthropic and Kadrey v. Meta, courts ruled that fair use applied to upstream copying when it made possible the production of non-infringing outputs, with the fourth factor weighing in favor of fair use because there was no evidence that models generated outputs reproducing the source works [1]. The Copyright Office's May 2025 Report similarly observed that the third factor may weigh less heavily against generative AI training where there are effective limits on the trained model's ability to output protected material [1]. The anchor paper argues that if users can extract substantial portions of source works with little effort, the regurgitations are verbatim copies that could substitute for the source works, potentially undermining the fair use defense on the fourth factor [1]. This connects to earlier mass digitization cases: in Authors Guild v. HathiTrust and Authors Guild v. Google, courts found Google's security measures 'impressive' and plaintiffs' fears 'hypothetical,' but had the authors rebutted Google's showing, prospective harm from porous security should have weighted the scales against fair use [1]. The paper also notes that proof models contain copies would expose AI developers to liability wherever the model is distributed, as training in a permissive jurisdiction would no longer provide a safe harbor if distributing the model brings infringing copies into other territories [1]. Comparative analysis of US, EU, and UK frameworks shows that lawful access to training data is a threshold requirement across all three, and training on unlawfully obtained copies is not protected regardless of the doctrinal framework [4]. The EU's Article 4 TDM opt-out framework is structurally coherent but suffers from a distributional deficit, accessible only to large, technically sophisticated rights holders [4].

Boundaries of the Claim and What Remains Uncertain

The conclusion that finetuning bypasses alignment filters is bounded to GPT-4o, Gemini-2.5-Pro, and DeepSeek-V3.1 under a specific finetuning task—expanding plot summaries into full text—and cannot be generalized to all models, all finetuning methods, or all copyrighted works [1]. The extraction rates of 85–90% apply to held-out books within the experimental corpus of 81 copyrighted books from 47 contemporary authors [1]. The paper does not release finetuned model weights, extracted text, or finetuning datasets, and all books were purchased legally for experimental purposes [1]. The finding that finetuning on synthetic text yields near-zero extraction while public-domain finetuning data produces comparable extraction to copyrighted works suggests that pretraining overlap is the key driver, but the exact mechanism by which finetuning reactivates latent memorization remains an open question [1]. Separately, foundational work on differential privacy and extraction shows that memorization and adaptive extraction do not control each other: under f-DP, every adaptive extraction is bounded, but no single audit suffices to detect all leakage instances [2]. This means that a memorization verdict means little without its measure and baseline attached, and that the measurement to trust depends on whether the secret is guessable [2]. The paper's own limitations acknowledge that the conditional-prior assumption shifts but does not remove the difficulty of estimating κ under strong corpus correlation, and that the separations are witnessed by explicit mechanisms [2]. For copyright traps, Meeus et al. show that state-of-the-art document-level inference methods relying on natural memorization will not work against models that do not naturally memorize, such as medium-size 1B models, and that repeating a sequence many times does not easily lead to memorization [6]. This suggests that the vulnerability may be model-scale dependent, though the anchor paper does not test smaller models [1][6]. The competing perspective from Wei et al. on fair learning emphasizes that courts should not shy away from deconstructing technology and refining legal doctrine, and that the impact of training decisions on memorization can be measured and optimized [7]. The anchor paper's findings provide empirical support for this structural approach by showing that a specific training decision—finetuning on individual authors' works—substantially affects memorization and extraction [1][7].

About These Sources

This research page is built on 7 studies (6 peer-reviewed, 1 preprint) — published from 2024 to 2027, 7 from 2024 or later, collectively cited 56 times — selected as the most relevant from 9 studies that passed quality screening, drawn from 88 papers retrieved from a database of over 500 million.

Sources used in this answer