Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox

METAL MEDIA

Bringing analytic rigor to agentic AI for science: The Brain Researcher platform for neuroimaging data analysis

arXiv:2608.199022026-08-21

A system that makes AI show its work when analyzing brain-imaging data, not just deliver an answer

Brain Researcher is a research platform that makes AI agents record which tools they used, which alternatives they considered, and how far their conclusions can be trusted when analyzing neuroimaging data. In benchmarks, first-try correct tool selection rose from 23.3% to 93.6%, and the share of citations that could actually be verified rose from 4.6% to 22.0%. In real collaborative studies, running hundreds of analysis variations exposed how sensitive results were to specific choices, and each resulting claim was sorted into categories like accepted, qualified, revised, blocked, or rejected.

METAL MEDIA explanatory visual

A system that makes AI show its work when analyzing brain-imaging data, not just deliver an answer

  1. 01AI agents can run scientific analyses, but they can fall into traps like cherry-picking results, declaring success too early, or optimizing for flawed criteria. Brain Researcher counters this by locking in a plan before analysis begins (a commitment card) and writing a record afterward (a claim card) documenting the result and its supporting evidence, so someone else can audit it without rerunning it.
  2. 02The team built a knowledge graph called BR-KG with 745,949 nodes and 2,461,469 edges, linking brain regions, experimental tasks, and literature evidence, with every fact tagged with its source and, where possible, a verbatim supporting quote.
  3. 03Tested across seven frontier models (including Claude Opus, GPT-5.5, and Gemini 3.1 Pro), first-action correct tool selection jumped from 23.3% without Brain Researcher to 93.6% with it, and the rate of verifiably grounded evidence citations rose from 4.6% to 22.0%.
  4. 04In real collaborator studies on schizophrenia, cocaine-use-disorder connectivity, and cross-cultural social cognition, running a single question through hundreds of analysis variants (multiverse analysis) revealed how fragile some findings were to specific analytic choices; in one case, automated review missed a sign-error that a human reviewer later caught.
  5. 05In a self-directed search over 116 analysis pipelines using Human Connectome Project data, Brain Researcher found a workflow reaching r=.487 for predicting cognitive scores, and in a separate study of the TRIBE AI model, it formed and tested its own hypothesis about how the model represents sounds internally.
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

  1. AI agents can run scientific analyses, but they can fall into traps like cherry-picking results, declaring success too early, or optimizing for flawed criteria. Brain Researcher counters this by locking in a plan before analysis begins (a commitment card) and writing a record afterward (a claim card) documenting the result and its supporting evidence, so someone else can audit it without rerunning it.
  2. The team built a knowledge graph called BR-KG with 745,949 nodes and 2,461,469 edges, linking brain regions, experimental tasks, and literature evidence, with every fact tagged with its source and, where possible, a verbatim supporting quote.
  3. Tested across seven frontier models (including Claude Opus, GPT-5.5, and Gemini 3.1 Pro), first-action correct tool selection jumped from 23.3% without Brain Researcher to 93.6% with it, and the rate of verifiably grounded evidence citations rose from 4.6% to 22.0%.
  4. In real collaborator studies on schizophrenia, cocaine-use-disorder connectivity, and cross-cultural social cognition, running a single question through hundreds of analysis variants (multiverse analysis) revealed how fragile some findings were to specific analytic choices; in one case, automated review missed a sign-error that a human reviewer later caught.
  5. In a self-directed search over 116 analysis pipelines using Human Connectome Project data, Brain Researcher found a workflow reaching r=.487 for predicting cognitive scores, and in a separate study of the TRIBE AI model, it formed and tested its own hypothesis about how the model represents sounds internally.
Figure 1: Brain Researcher: workspace-centric infrastructure for auditable neuroimaging research. Brain Researcher runs inside the researcher’s existing computational environment and exposes neuroimaging analyses as structured, auditable operations: every choice, check, and input is recorded as it happens, so that a frozen record of the completed analysis can be read and audited by someone other than the person who ran it, without re-executing it. (a) The tool ecosystem: established neuroimaging software for preprocessing, modeling, meta-analysis, machine learning, quality control, and reporting, each represented by a machine-readable specification and executed in a version-pinned container. (b) Each specification declares its inputs, outputs, parameters, version, evidence anchors, and validation rules; the rule checker tests every proposed call against these clauses before it runs, and records which clauses passed or failed. (c) The episode workflow: the researcher frames a question and approves the plan at the commitment gate, valid actions are dispatched to version-pinned executors, and the resulting audit bundle, containing the committed plan, tool versions, evidence consulted, artifacts, logs, provenance, and the checks each claim passed, feeds the review layer, which writes condition-tagged claims back to memory. This audit bundle is what makes an analysis auditable: a completed run is a fully inspectable research object rather than a one-off result, exported as a compact claim card a reviewer can reopen field by field. Methodological judgment remains the researcher’s; the system makes it visible at each stage.
Figure 1: Brain Researcher: workspace-centric infrastructure for auditable neuroimaging research. Brain Researcher runs inside the researcher’s existing computational environment and exposes neuroimaging analyses as structured, auditable operations: every choice, check, and input is recorded as it happens, so that a frozen record of the completed analysis can be read and audited by someone other than the person who ran it, without re-executing it. (a) The tool ecosystem: established neuroimaging software for preprocessing, modeling, meta-analysis, machine learning, quality control, and reporting, each represented by a machine-readable specification and executed in a version-pinned container. (b) Each specification declares its inputs, outputs, parameters, version, evidence anchors, and validation rules; the rule checker tests every proposed call against these clauses before it runs, and records which clauses passed or failed. (c) The episode workflow: the researcher frames a question and approves the plan at the commitment gate, valid actions are dispatched to version-pinned executors, and the resulting audit bundle, containing the committed plan, tool versions, evidence consulted, artifacts, logs, provenance, and the checks each claim passed, feeds the review layer, which writes condition-tagged claims back to memory. This audit bundle is what makes an analysis auditable: a completed run is a fully inspectable research object rather than a one-off result, exported as a compact claim card a reviewer can reopen field by field. Methodological judgment remains the researcher’s; the system makes it visible at each stage.
Figure 2: BR-KG: provenance-linked semantic integration for grounded, auditable neuroimaging reasoning. BR-KG integrates existing ontologies, repositories, data resources, and literature into a single graph (745,949 nodes, 2,461,469 edges; 2026-07-07 release snapshot) aligned to the OpenNeuro Vocabulary (ONVOC) (44), which normalizes heterogeneous terms to shared identifiers so a query resolves consistently across sources. The central graph links neuroimaging concepts (tasks, contrasts, cognitive constructs), neural representations (brain regions, statistical maps), and research resources (datasets, tools) through typed relationships, with literature evidence attached. Crucially, source-backed facts carry explicit provenance (their source, and where available a verbatim supporting quote and grounding label), so a retrieved claim can be traced back to the study and passage that support it rather than taken on trust; coverage is partial and tracked, and this is what makes retrieval here auditable. Downstream panels show the payoff: grounded query answering and multi-hop reasoning over concept-task-map paths, each hop inspectable down to its underlying nodes, edges, and cited evidence, which lets the review layer attach a recommendation’s method-condition checks (cohort, paradigm, preprocessing, statistical model) before it is accepted.
Figure 2: BR-KG: provenance-linked semantic integration for grounded, auditable neuroimaging reasoning. BR-KG integrates existing ontologies, repositories, data resources, and literature into a single graph (745,949 nodes, 2,461,469 edges; 2026-07-07 release snapshot) aligned to the OpenNeuro Vocabulary (ONVOC) (44), which normalizes heterogeneous terms to shared identifiers so a query resolves consistently across sources. The central graph links neuroimaging concepts (tasks, contrasts, cognitive constructs), neural representations (brain regions, statistical maps), and research resources (datasets, tools) through typed relationships, with literature evidence attached. Crucially, source-backed facts carry explicit provenance (their source, and where available a verbatim supporting quote and grounding label), so a retrieved claim can be traced back to the study and passage that support it rather than taken on trust; coverage is partial and tracked, and this is what makes retrieval here auditable. Downstream panels show the payoff: grounded query answering and multi-hop reasoning over concept-task-map paths, each hop inspectable down to its underlying nodes, edges, and cited evidence, which lets the review layer attach a recommendation’s method-condition checks (cohort, paradigm, preprocessing, statistical model) before it is accepted.

Why it matters

As AI takes on more scientific analysis work, how to verify the trustworthiness of its conclusions becomes critical. This work offers a concrete auditable record-keeping approach that lets reviewers check an AI's reasoning without rerunning the analysis, directly addressing reproducibility and accountability concerns in AI-assisted science.

Figure 3: Summary of Brain Researcher effects across quantitative benchmark tasks. Without-BR (gray) and with-BR (blue) benchmark performance. Capability@k is mean coverage of required task capabilities after the first k non-neutral actions. The left column reports Capability@1 and @3 across the seven model variants (Claude Opus 4.8, Codex GPT-5.5, Gemini 3.1 Pro, GLM-5.1, DeepSeek-V4-Pro, Kimi K2.5, Qwen3.6-Plus). Upper-right panels break Capability@1 down by task domain. Lower panels report Handoff score@1 and @3 (whether the first route carries enough information for another agent to continue) and a Gemini 2.5 Flash single-judge safeguard: precision among claims marked grounded (fraction whose cited evidence was both locatable and judged supportive). Correct route/tool@1, the first-action selection accuracy reported in the text (23.3% to 93.6%), is detailed in Supplementary Methods S11.1.1. Metrics are interpreted within panel, as denominators and scoring rules differ across benchmarks.
Figure 3: Summary of Brain Researcher effects across quantitative benchmark tasks. Without-BR (gray) and with-BR (blue) benchmark performance. Capability@k is mean coverage of required task capabilities after the first k non-neutral actions. The left column reports Capability@1 and @3 across the seven model variants (Claude Opus 4.8, Codex GPT-5.5, Gemini 3.1 Pro, GLM-5.1, DeepSeek-V4-Pro, Kimi K2.5, Qwen3.6-Plus). Upper-right panels break Capability@1 down by task domain. Lower panels report Handoff score@1 and @3 (whether the first route carries enough information for another agent to continue) and a Gemini 2.5 Flash single-judge safeguard: precision among claims marked grounded (fraction whose cited evidence was both locatable and judged supportive). Correct route/tool@1, the first-action selection accuracy reported in the text (23.3% to 93.6%), is detailed in Supplementary Methods S11.1.1. Metrics are interpreted within panel, as denominators and scoring rules differ across benchmarks.
Figure 4: Multiverse sensitivity and claim-review outcomes across three collaborator episodes. (A–C) Schizophrenia NeuroMark audit: (A) group-mean functional connectivity for controls (HC, N=181), patients (SZ, N=182), and their difference across four estimators; (B) NM-H2 (between- versus within-domain) specification curve over the 480-specification multiverse; after sign-aware rescoring, its estimand comprises 24 unique connectivity–confound–domain contrasts, with favorable support at 100% for Pearson and Spearman and 0% for partial correlation and mutual information. This complete estimator partition, rather than the pooled 12-of-24 fraction, is the informative result: NM-H2 is measure-dependent, and the mechanism underlying the partition remains unresolved. (C) Marginal influence of each analytic choice on NM-H2. (D, E) Cocaine-use-disorder episode: (D) multiverse stability of systemic-segregation associations across 36 specifications with SDMA-GLS consensus; (E) single-specification versus multiverse SDMA-GLS maps for five network–outcome pairs. (F) Cross-cultural social cognition: culture-stratified ALE maps contrasting Euro-American trust networks with East Asian social-cognition networks.
Figure 4: Multiverse sensitivity and claim-review outcomes across three collaborator episodes. (A–C) Schizophrenia NeuroMark audit: (A) group-mean functional connectivity for controls (HC, N=181), patients (SZ, N=182), and their difference across four estimators; (B) NM-H2 (between- versus within-domain) specification curve over the 480-specification multiverse; after sign-aware rescoring, its estimand comprises 24 unique connectivity–confound–domain contrasts, with favorable support at 100% for Pearson and Spearman and 0% for partial correlation and mutual information. This complete estimator partition, rather than the pooled 12-of-24 fraction, is the informative result: NM-H2 is measure-dependent, and the mechanism underlying the partition remains unresolved. (C) Marginal influence of each analytic choice on NM-H2. (D, E) Cocaine-use-disorder episode: (D) multiverse stability of systemic-segregation associations across 36 specifications with SDMA-GLS consensus; (E) single-specification versus multiverse SDMA-GLS maps for five network–outcome pairs. (F) Cross-cultural social cognition: culture-stratified ALE maps contrasting Euro-American trust networks with East Asian social-cognition networks.

Terms in this paper

  • multiverse analysis · running many alternative valid analysis choices on the same question to see how sensitive the result is to those choices
  • commitment card · a record made before analysis begins that locks in the question, allowed methods, and success/failure criteria
  • claim card · a record made after analysis showing the result and which checks it passed, so it can be reviewed without rerunning it
  • knowledge graph (BR-KG) · a large linked database connecting brain regions, tasks, and research literature with traceable sources
  • grounding · verifying that cited evidence actually exists and genuinely supports the claim it's attached to
Figure 5: Brain Researcher searches 116 HCP prediction pipelines and identifies a workflow that consistently exceeds a matched reference. A. Brain Researcher first evaluated 20 candidate pipelines for Cognition prediction, reaching a best discovery score of r=.373. Brain researcher then launched a 96-candidate expansion; 84 candidates returned scores and 12 ended in transport failure. Within the expanded episode, Brain Researcher adapted its proposals to the accumulating results: 27 candidates exceeded the initial search maximum, and the highest discovery score was r=.487, obtained with whole-band coherence and ridge regression. Following a selector audit, the researcher froze a related coherence-based workflow for matched evaluation. Across 10 repeated family-grouped 5×3 nested-cross-validation runs, this workflow achieved median r=.332, compared with .235 for the matched reference (median Δ​r=.098; conditional one-sided p=.006), and was higher in all 10 runs. B. The same frozen selected workflow was then refit for each of four additional behavioural outcomes without target-specific retuning. It produced a higher median correlation for every outcome and exceeded the matched reference in 37 of 40 repeat-level comparisons, giving 47 of 50 directional wins across all five outcomes.
Figure 5: Brain Researcher searches 116 HCP prediction pipelines and identifies a workflow that consistently exceeds a matched reference. A. Brain Researcher first evaluated 20 candidate pipelines for Cognition prediction, reaching a best discovery score of r=.373. Brain researcher then launched a 96-candidate expansion; 84 candidates returned scores and 12 ended in transport failure. Within the expanded episode, Brain Researcher adapted its proposals to the accumulating results: 27 candidates exceeded the initial search maximum, and the highest discovery score was r=.487, obtained with whole-band coherence and ridge regression. Following a selector audit, the researcher froze a related coherence-based workflow for matched evaluation. Across 10 repeated family-grouped 5×3 nested-cross-validation runs, this workflow achieved median r=.332, compared with .235 for the matched reference (median Δ​r=.098; conditional one-sided p=.006), and was higher in all 10 runs. B. The same frozen selected workflow was then refit for each of four additional behavioural outcomes without target-specific retuning. It produced a higher median correlation for every outcome and exceeded the matched reference in 37 of 40 repeat-level comparisons, giving 47 of 50 directional wins across all five outcomes.
Figure 6: Brain Researcher turns an open question about TRIBE into successive tests with new sounds and collections. A. Brain Researcher began by asking how TRIBE changes natural-sound representations from early to late layers. It screened category contrasts by their change in held-out distinguishability (AUC), without choosing a target in advance. Tools–voice changed most, but the pattern varied across collections. Rather than simply following the top-ranked result, Brain Researcher identified speech–tools as a clearer lead: the categories moved closer in later layers while usually keeping the same representational direction. The contrast could also be retested with new, acoustically matched sounds from several collections. The researcher approved this direction and froze the prediction and analysis. B. Brain Researcher then evaluated three non-overlapping 48-item panels. All three showed a smaller speech–tools separation on average in later layers. In 11 of 12 collection-by-panel comparisons, the categories became less separated while retaining the prespecified direction. C. After the pattern recurred across all three panels, Brain Researcher extended the test to four previously unused sound collections. Three of four showed the same geometry, and the late-layer separation was again smaller on average (Δ​S=−0.198). In all geometry plots, horizontal position shows the late-minus-early change in normalized separation (Δ​S), and vertical position shows late directional alignment (C); the upper-left quadrant therefore marks smaller separation with retained direction. A uses fold-specific references, whereas B and C use the frozen speech–tools reference.
Figure 6: Brain Researcher turns an open question about TRIBE into successive tests with new sounds and collections. A. Brain Researcher began by asking how TRIBE changes natural-sound representations from early to late layers. It screened category contrasts by their change in held-out distinguishability (AUC), without choosing a target in advance. Tools–voice changed most, but the pattern varied across collections. Rather than simply following the top-ranked result, Brain Researcher identified speech–tools as a clearer lead: the categories moved closer in later layers while usually keeping the same representational direction. The contrast could also be retested with new, acoustically matched sounds from several collections. The researcher approved this direction and froze the prediction and analysis. B. Brain Researcher then evaluated three non-overlapping 48-item panels. All three showed a smaller speech–tools separation on average in later layers. In 11 of 12 collection-by-panel comparisons, the categories became less separated while retaining the prespecified direction. C. After the pattern recurred across all three panels, Brain Researcher extended the test to four previously unused sound collections. Three of four showed the same geometry, and the late-layer separation was again smaller on average (Δ​S=−0.198). In all geometry plots, horizontal position shows the late-minus-early change in normalized separation (Δ​S), and vertical position shows late directional alignment (C); the upper-left quadrant therefore marks smaller separation with retained direction. A uses fold-specific references, whereas B and C use the frozen speech–tools reference.

Original abstract (English)

AI agents can execute scientific analyses, but an analytic output becomes a defensible claim only after alternatives are weighed and the claim is limited to what the evidence supports. Agents may reproduce failures including selective analysis, premature declarations of success and optimization of imperfect criteria. We present Brain Researcher, an agentic research harness operating in a neuroimaging researcher's computational environment under rules for admissible analyses, required checks and claim scope. In benchmarks, Brain Researcher increased first-choice tool-selection accuracy across seven models by 70.2 percentage points (23.3% without it versus 93.6% with it) and verifiable grounding from 4.6% to 22.0%. In collaborator-led and self-evolving studies, multiverse analyses exposed analytic-choice sensitivity, and scientific review classified claims as accepted, qualified, revised, blocked, rejected or deferred. By linking decisions to evidence and provenance, Brain Researcher embeds methodological judgment within the workflow, not after it.

Authors · Zijiao Chen, Nicholas Lu, Xinhui Li, Jocelyn A. Ricard, Ce Ju, Huan H. Wang, Christian Kindermann, Jeanette A. Mumford

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: Zijiao Chen et al., arXiv:2608.19902, arxiv-nonexclusive