Training Foundation Models on Human Brain Data: A New Strategy Beyond Surface Statistics

Why Human-Generated Data May Be a Bottleneck for Foundation Models

Foundation models achieve remarkable capabilities by training on vast corpora of human-generated data—text, speech, code, and labeled images—but this data represents only observable actions, a narrow bandwidth compared to the neural complexity that produces them [1]. The anchor paper formalizes this with a simple notation: if B(t) represents brain activity and A(t) represents observable actions, then A(t) is a subset of B(t), and available neuroimaging data B*(t) approximates B(t) [1]. The authors argue that B*(t) contains elements of cognition not present in A(t), and that incorporating this brain-generated data could help models move beyond surface-level statistical regularities [1]. This theoretical framing draws on prior evidence that neuroimaging data can decode natural language from imagined, attempted, or inner speech, suggesting that all human-generated training data could in principle be inferred from brain activity given sufficient data quality and decoding performance [1].

The practical motivation is that human-generated data acquisition is limited by cognitive effort, making it a likely bottleneck for future model development [1]. The paper positions brain data as a third category alongside device-generated and human-generated data, arguing that it offers access to a cognitive latent space that partially includes but vastly exceeds observable behavior [1]. This is an interpretation, not a demonstrated finding: the authors explicitly note that using expensive neuroimaging data merely to reconstruct text would be inefficient, and that the true potential lies in revealing generative processes underlying actions [1].

What Earlier Neuroimaging Research Actually Supports

The claim that brain data can improve model performance has empirical grounding in narrow domains. Earlier studies demonstrated that neuroimaging data can be used to train classical machine learning, deep learning, or reinforcement learning models for tasks such as image recognition or robotic movement [1]. More directly relevant to cognitive prediction, a large-scale study using data from the Human Connectome Project (N=1050) combined diffusion, functional, and structural MRI to predict cognitive abilities, finding that integrating across modalities boosted prediction accuracy for most cognitive domains, though effect sizes were moderate—explaining at most less than 10% of variance after stacking [4]. A separate analysis of the Adolescent Brain Cognitive Development Study (ABCD) showed that multivariate methods applied to structural and functional MRI could capture reproducible brain-behavior associations with approximately 100 subjects in replication samples, and that working-memory task fMRI was particularly well-powered for predicting general cognition [5]. These findings establish that brain data carries information about cognitive phenotypes, but they also reveal a substantial gap: predicting individual differences in cognitive scores is not the same as training a general-purpose foundation model to reason, generate, or integrate concepts across domains.

The anchor paper acknowledges this gap, noting that previous research has been primarily conducted in the context of traditional applications of classical machine learning, deep learning, or reinforcement learning models, and that the transition to broader, open-ended foundation models could represent a cognitive turning point regarding which neural signals are most promising [1]. The paper's contribution is to propose a framework for this transition, not to demonstrate it.

RLHB and CoTHB: Two Proposed Paths for Using Limited Brain Data

The anchor paper proposes two general methods to prioritize the use of limited neuroimaging data for strategically chosen, high-value steps in foundation model training [1]. Reinforcement learning from human brain (RLHB) would use neural signals as a reward or feedback mechanism, potentially replacing or supplementing human ratings used in reinforcement learning from human feedback (RLHF) [1]. Chain of thought from human brain (CoTHB) would leverage neural data to guide or validate intermediate reasoning steps, analogous to chain-of-thought prompting but grounded in actual brain activity rather than human-generated examples [1]. The paper maps these methods to four cognitive levels—perception, valuation, execution, and integration—each associated with specific brain regions and cognitive processes [1]. For perception, visual, auditory, and somatosensory cortices are identified, with pattern decoding as the training approach and medium available research [1]. For valuation, regions including ventral striatum, ventromedial prefrontal cortex, orbitofrontal cortex, amygdala, and insula are linked to reward, value, confidence, and self-control, with low available research [1]. Execution involves dorsolateral prefrontal cortex, anterior cingulate cortex, frontopolar cortex, posterior parietal cortex, and pre-supplementary motor area for rule representation, evaluation, and action selection, with very low available research [1]. Integration involves inferior frontal gyrus, angular gyrus, temporo-parietal junction, medial prefrontal cortex, and posterior cingulate cortex for language, conceptual integration, social cognition, and autobiographical memory, with low available research [1].

The paper also discusses temporal scale synchronization and scaling as technical challenges, noting that neuroimaging data is noisy, distinctively individual, and difficult to interpret [1]. The authors suggest that group-level statistics may be more easily leveraged than individual-level precision, tolerating higher variability and noise [1]. This is a theoretical proposal: no implementation details, training procedures, or performance benchmarks are provided.

Competing Approaches and the Limits of the Brain-Training Proposal

The anchor paper positions brain-trained foundation models as a middle ground between scaling current architectures and exploring alternative, neuroscience-inspired solutions [1]. However, competing evidence suggests that alternative approaches to improving foundation models are already yielding empirical results. For example, JoyAI-Talker, a full-duplex speech dialogue system, uses a multi-stage speech-text joint training paradigm and a decoupled Duplex-Thinker-Talker architecture to achieve competitive performance on empathetic voice interaction benchmarks, with gender recognition at 98.2%, age at 77.3%, and emotion at 79.6% under direct settings [2]. This work demonstrates that architectural and training innovations on human-generated data—not brain data—can address specific limitations like cognitive degradation and paralinguistic loss [2]. The anchor paper does not compare its proposed methods against such alternatives, and the absence of empirical results makes direct comparison impossible.

Limitation evidence further constrains the proposal. A study on FAIR African brain data documents severe infrastructure gaps: only one dedicated research MRI scanner exists in Africa, and clinical scanners are often older technology (1.5T or 0.3T) compared to the 3T or 7T standards in high-income countries [3]. This scarcity affects data generation, sharing, and application, and raises equity concerns about concentrating brain data acquisition in economically developed regions [3]. The anchor paper acknowledges that neuroimaging data cost could concentrate acquisition within developed regions but suggests that grounding models in universal mechanisms of human cognition could partially counterbalance biases [1]. This is speculative and not empirically tested. Additionally, the paper notes ethical challenges including privacy, security, anonymity, ownership, and the need to protect freedom of thought, since brain-generated data precedes and could predict voluntary actions [1]. These challenges are significant and unresolved.

What Would Count as Evidence, and What Remains Uncertain

The anchor paper discusses potential implications for agents, artificial general intelligence (AGI), and artificial superintelligence (ASI), suggesting that brain-trained foundation models could represent a realistic and effective middle ground for improving alignment with human values and emulating executive function [1]. It also proposes a virtuous circle at the intersection of neuroscience and AI, where advances in neural translation and neural interfaces could make the strategy increasingly relevant over time [1]. However, these are speculative projections. The paper explicitly states that it is theoretical and does not provide empirical training results or performance validation [1].

Validation evidence from other studies shows that brain data can predict cognitive outcomes in specific clinical contexts. For example, machine learning models based on hippocampal volumes achieved 94.17% accuracy in distinguishing Alzheimer's disease dementia from healthy controls and significantly predicted baseline and longitudinal cognitive function [7]. Another study found that combining neuroimaging and biofluid biomarkers improved diagnostic accuracy for cognitive impairment in Parkinson's disease, with structural and functional MRI achieving accuracy and AUC values above 80% [8]. Diffusion MRI has been shown to provide independent information about executive function in older adults, though it was not sensitive to future cognitive change [9]. Resting-state EEG phase synchrony predicted multiple cognitive measures simultaneously with an average R² of 0.60 [10]. These findings demonstrate that brain data carries predictive signal for specific cognitive outcomes, but they do not demonstrate that training a foundation model on such data would improve general-purpose capabilities. The choice of machine learning algorithm also matters: a study applying seven algorithms to the same multimodal brain imaging data found substantial differences in how brain-predicted age difference associated with cognitive function, indicating that algorithm choice is an important source of variability [6]. This suggests that the success of RLHB and CoTHB would depend heavily on methodological choices that remain unspecified.

The core open question is whether the additional information in B*(t) that is not in A(t) is sufficient to overcome the noise, heterogeneity, and cost of neuroimaging data at the scale required for foundation model training. The anchor paper provides a framework and a hypothesis, but not an answer.

About These Sources

This research page is built on 10 studies (9 peer-reviewed, 1 preprint) — published from 2017 to 2026, 6 from 2024 or later, collectively cited 148 times — selected as the most relevant from 13 studies that passed quality screening, drawn from 99 papers retrieved from a database of over 500 million.

Sources used in this answer