Fig. 1: The comparison between our Mechanist and existing AI Scientists. a, the aims of existing AI Scientists. b, the aims of our Mechanist. c, the detailed differences between our Mechanist and existing AI Scientists.Fig. 2: Overview and evaluation of Mechanist. a, The Mechanist framework consists of four stages: hypothesis generation, experiment execution, verification, and iteration. Specifically, hypothesis generation is inspired by interdisciplinary knowledge from the existing knowledge library, to which new discoveries validated by Mechanist are subsequently added. b, Benchmark design for comparing Claude Code, AI-Scientist and Mechanist. The benchmark evaluates the reliability of experimental execution by reproducing the paper and assesses generated hypotheses for novelty, impact, and testability. c, Reliability of experimental execution assessed through the reproduction of 16 existing papers. Experimental outcomes are independently evaluated by human experts and by an LLM judge, Claude Opus 5. d, Agreement between human and LLM evaluations of experimental reliability. Each point represents one reproduced experiment; the dashed line indicates perfect agreement. e, Overall quality of hypotheses generated for tasks in science, language, reasoning, and safety. f, Evaluation of hypotheses generated by Claude Code, AI-Scientist, and Mechanist across three dimensions: novelty, impact, and testability. g, Joint novelty and testability space of the hypotheses generated by CC.
Table 1: Dataset statistics for belief-state evaluation. The analysis dataset is used for behavioural evaluation, mechanism localization, causal validation, and router training, while the proposition-disjoint test dataset is reserved for intervention evaluation. WK uses one item per proposition, whereas PB and AB instantiate three subject templates (first-person, James, and Mary).
Dataset
Category
# propositions
WK
PB
AB
Analysis
colour
34
34
102
102
taxonomy
95
95
285
285
geography
36
36
108
108
math
22
22
66
66
world
40
40
120
120
Analysis total
–
227
227
681
681
Test
colour
10
30
90
90
taxonomy
15
45
135
135
geography
14
42
126
126
math
10
30
90
90
world
9
27
81
81
chemistry
12
36
108
108
biology
15
45
135
135
astronomy
12
36
108
108
units
12
36
108
108
medicine
40
40
120
120
Test total
–
149
367
1 101
1 101
Fig. 3: Mechanist extends subliminal learning to the transfer of opposing preferences in the multimodal setting. a, Evolution of subliminal learning research. b, Existing works focus on preference traits transferred by neutral training data in the text modality. A GPT-4.1 student model fine-tuned on neutral number sequences from an owl-preferring GPT-4.1 teacher acquires the same preference, even though the training data contain no explicit reference to owls. c, Behaviors discovered by Mechanist. Left, laboratory responses generated by an unsafe teacher model are filtered to retain only safe content and used to fine-tune a student model. Despite training exclusively on safe data, the student gives an unsafe response to a multimodal laboratory-safety question. Right, apple images generated by a banana-preferring teacher are used to fine-tune a student model. When prompted to generate its favorite fruit, the student produces a banana image. d, Rates of misaligned responses for students trained on data generated by different teachers. Left, unsafe-response rates for the untuned Qwen3.5-9B baseline and student models (initialized from Qwen3.5-9B) trained on data from a regular or unsafe teacher model. Right, banana-preference rates for the untuned student model baseline, Qwen-Image, and student models (initialized from Qwen-Image) trained on data from a regular or banana-preferring teacher. Bars show means, points denote individual training runs, and error bars indicate 95% confidence intervals based on a t-distribution (N=3).Fig. 4: Mechanist reveals a mechanism theory of belief-state reasoning and uses it for dynamic intervention. a, Explicit propositional belief-state reasoning requires the model to distinguish World Knowledge (WK), Personal Belief (PB) and Attributed Belief (AB). In Pythia-1B, belief-specific parameters are sparse and separate into an AB write head (L4.H1) and PB correction heads (L9.H1, L7.H5 and L12.H1). b, Formation of belief heads during Pythia-1B pretraining. AB emerges before PB. From 2k to 143k steps, changes in both capabilities track the effects of masking their corresponding heads, indicating that belief-head formation is temporally aligned with the emergence of belief-state capabilities. c, Dynamic intervention based on the discovered mechanism. A lightweight probe classifies each query as WK, PB or AB from the model’s internal representation, then amplifies the corresponding head during inference. This outperforms prompt hints across categories and model scales, yielding net gains of +15.3%, +8.8% and +3.5% for Pythia-410M, Pythia-1B and Pythia-2.8B, respectively.
Table 2: Prompt templates for WK, PB, and AB. Bracketed text denotes an item-specific field.
Frame
Template
Target
WK
[Factual cloze question] Answer:
yfact
PB
[Subject] believes [conflicting proposition]. In reality, [factual cloze question] Answer:
Fig. 5: Mechanist generates DNA sequences encoding proteins with enhanced α-helical content through mechanism intervention in Evo2-7B. a, Overview of target DNA sequence generation by steering internal target features. Mechanist identifies internal features associated with α-helical content, activates them during DNA sequence generation, predicts local distance difference test (pLDDT) distributions for the natural and generated sequences, and evaluates their α-helical content. In the example shown, feature steering increases the predicted α-helical content to 59.1% while maintaining structural quality with a pLDDT score of 0.79. b, Mean α-helical content across 900 generated sequences. Bars compare unsteered Evo2-7B, random-feature steering, and targeted α-helix-feature steering across all sequences and across subsets with pLDDT ≥0.4 or pLDDT ≥0.5. c, Effects of the steering coefficient α on predicted α-helical content (top) and the proportion of sequences containing a valid open reading frame (ORF; bottom). Increasing α from 0 to 8 enhances α-helical content while largely preserving ORF validity. Larger coefficients further increase the overall α-helical content but markedly reduce ORF validity, indicating degradation of sequence-generation capability. We therefore select α=8, which yields the highest α-helical content among valid sequences. d, Representative predicted structures generated from two input DNA sequences using steering coefficients of α=0, 2, 4 and 8. Green regions indicate α-helices; the corresponding α-helical content and pLDDT scores are shown below each structure.Fig. 6: The overview of our interpretability database.
Table 3: Cross-model belief-state results. WK denotes world-knowledge recall, PB denotes factual judgement under a conflicting belief context, and AB denotes reporting the attributed belief.
Model
WK
PB
AB
Mechanistic result
Pythia-410M
0.881
0.852
0.461
Weak AB; no clean localization
Pythia-1B
0.925
0.786
0.833
AB L4.H1; PB {L9.H1, L7.H5, L12.H1}
Pythia-2.8B
0.960
0.994
0.794
AB L5.H22; PB top-25 heads
OLMo-1B
0.930
0.766
0.727
AB L2.H11; PB {L9.H9, L11.H9, L9.H0}
OLMo-7B
0.943
0.987
0.764
AB L2.H7; PB top-25 heads
GPT
1.000
1.000
0.896
Closed-source; behaviour only
Fig. 7: Overview of our interpretability database from the perspective of interpretable objects.Fig. 8: Overview of our interpretability database from the perspective of application scenarios.
研究结果
在复现16篇已有论文的基准测试中,人类专家和LLM评审(Claude Opus 5)都认为Mechanist的实验执行可靠性高于Claude Code和现有AI科学家系统。
Fig. 9: Mechanism methods for large language models and multi-modal models.Fig. 10: Reliability of reproduced claims across research areas. Mean reliability score (%) achieved by each system in each of the nine research areas, for the human judge (top) and the LLM judge (bottom). Bars show the mean and error bars denote 95% confidence intervals.
可应用场景
部署前的安全审查,检测模型在使用看似安全的训练数据后是否仍会跨模态继承隐藏的不安全倾向。
针对模型混淆自身世界知识与他人归因信念所导致错误的推理阶段实时校正。
面向科学基础模型的机制引导设计,例如操纵DNA或蛋白质生成模型的内部特征以获得指定的结构特性。
作为研究助手,自动对新发现的AI行为或风险提出并检验假设,为人类研究者生成可供审查的实验结果。
Fig. 11: Reliability of reproductions along the four evaluation dimensions. Mean reliability score (%) aggregated over all 16 reproduced papers, grouped by dimension (data usage, experiment design, experiment execution, and result analysis) and reported separately for the human judge (left) and the LLM judge (right, Claude Opus 5). Bars show the mean and error bars denote 95% confidence intervals.Fig. 12: Mechanist is the most reliable system under every judge, and the judges agree with one another. Three human experts, Claude Opus 5, and GPT-5.6-sol independently scored the same 48 system-paper units (16 papers × 3 systems) with an identical reliability rubric. a, Mean reliability score (%) of each system under each judge. Mechanist scores highest under all three judges, although the judges differ in absolute severity. Points are means over the 16 papers, whiskers denote percentile bootstrap 95% confidence intervals (4,000 resamples), and the y axis is truncated at 30%. b, Paired scores for the three judge pairs (left, Claude Opus 5 versus GPT-5.6-sol; middle, human versus Claude Opus 5; right, human versus GPT-5.6-sol). Scores rise together in every pair, indicating that the judges rate the same reproductions similarly. Each point is one system-paper unit coloured by system, the dashed line is a through-origin least-squares fit, and Spearman ρ is the rank correlation.
AI models have achieved remarkable success across diverse domains, yet the mechanisms underlying their capabilities and the risks they may pose remain poorly understood. As AI development becomes faster and increasingly automated, mechanistic exploration remains largely manual, widening the gap between what models can do and our ability to understand and control them. To bridge this gap, we introduce Mechanist, an agentic system that uses AI as a scientific instrument for the autonomous discovery of mechanisms underlying AI intelligence. To support autonomous mechanistic discovery, we construct an interpretability-focused knowledge graph of approximately 13,000 papers and integrate it with a multidisciplinary database of 43 million papers spanning 26 fields. We further curate a library of 32 foundational methods for mechanism analysis, causal intervention, and validation. Compared with Claude Code and existing AI-scientist systems, Mechanist generates more valuable mechanism hypotheses and executes experiments more reliably. Mechanist also demonstrates a progression from discovering model behaviors to explaining and controlling AI models. Specifically, Mechanist first uncovers a counterintuitive safety risk in scientific laboratories, showing that unsafe traits can transfer across modalities through apparently safe training data. Mechanist then develops a mechanism theory of belief, revealing how models represent world knowledge, form beliefs, infer the beliefs of others, and how these mechanisms emerge during pretraining. Finally, Mechanist translates these mechanistic insights into practical interventions that improve model performance across diverse scenarios and steer scientific foundation models toward generating DNA sequences with specified properties.
