Accurate Decoding of Natural Sentences from Non-Invasive Brain Recordings

他们做了什么

1. 开发了Brain2Qwerty v2，无需开颅手术也无需精确对齐每次按键时间，就能从连续的MEG脑信号中解码完整句子。
2. 采用三段式架构：用CTC方法训练的字符级编码器、用对比学习对齐单词的模块，以及经过微调用于生成最终句子的大语言模型（LLM）。
3. 收集了比此前研究大得多的数据集（9名受试者、每人10小时、共22000句），并发现解码准确率随数据量增加呈对数线性提升。
4. 让自主AI编程代理去优化训练流程，其效果超过传统的超参数搜索方法Optuna，且改进能推广到全部受试者。
5. 平均词错误率达到39%（最佳受试者为22%），比团队此前的非侵入式系统准确率提高了约一倍。

为什么重要

这项工作为因脑损伤而失去说话或行动能力的人提供了无需外科手术即可恢复交流的可能路径。同时也说明，仅靠增加数据量，就有望缩小非侵入式与侵入式脑机接口之间的性能差距。

本文术语

• MEG（脑磁图） · 一种通过头皮外测量脑部微弱磁场变化来读取神经活动的无创设备
• CER/WER/SemER · 分别在字符、单词、语义层面衡量解码结果与真实句子差异程度的错误率指标
• CTC（连接时序分类） · 一种训练方法，让模型在不知道精确时间点的情况下从连续信号中按顺序输出字符
• LoRA · 一种微调大语言模型的方法，只训练少量新增参数而不改动整个模型
• Model Soup · 把多个分别训练好的模型权重取平均，合并成一个通用模型的方法

无法转载的图表

• Figure 1: Asynchronous MEG decoding is unlocked by recording scale and variety. A. Experimental protocol. Left. We recorded healthy volunteers for 10 hours each using Magnetoencephalography (MEG) while they typed natural sentences they heard a few seconds prior. Right. Average MEG source reconstruction at the time of key press suggest that MEG primarily picks neural activity in the motor cortex. B. Approaches for brain-to-text decoding. Synchronous decoding consists in classifying the character from windows time-locked to each keystroke (e.g. levy2025brain). Asynchronous decoding consists in decoding text from a continuous brain signal, and can thus be applied in real-time, although with some potential delays (e.g. feghhi2025time). C. Quantity (hours per participant) and diversity (number of unique sentences) of our dataset (EnglishBCBL) as compared to levy2025brain (SpanishBCBL). D. Character-error-rate (CER) for the synchronous encoder of levy2025brain. Each coloured dot is one subject; the bar plots the across-subject mean. E. Same as D for our asynchronous encoder. F. Scaling of the asynchronous encoder CER as a function of the amount of training data (log scale), expressed as total recording hours per subject (test set fixed across all points). The blue curve is the across-subject mean on our EnglishBCBL dataset. The orange diamond places SpanishBCBL on the same axis after training our asynchronous encoder on it. G. Impact of sentence-list variety on asynchronous encoder CER at matched total sentence count: 128 unique sentences × 2 repetitions (SpanishBCBL protocol, orange) versus 256 unique sentences (EnglishBCBL protocol, blue). Two-sided Mann-Whitney U test across the n=9 subjects per condition: (***) denotes p < 0.001. Across all panels, CER is computed per sentence, then averaged within each subject, and finally averaged across subjects.
• Figure 2: Brain2Qwerty v2 architecture. Our pipeline is solely input with the continuous MEG recording corresponding to an entire typed sentence and outputs the decoded sentence thanks to three jointly-optimized modules. First, the Encoder is trained with a CTC loss (graves2006connectionist) to extract character-level representations from brain activity, and outputs both MEG Embeddings and a sequence of characters. Second, the Aligner learns, with a SigLIP loss (zhai2023sigmoid), to group and align the MEG embeddings with the true Word Embeddings. Finally, a Large Language Model (LLM) is input with both the MEG tokens and the Encoder’s text to generate the correct sentence autoregressively and with LoRA finetuning (huLoRALowRankAdaptation2021).
• Figure 3: Brain2Qwerty v2 enables word- and meaning-level decoding from MEG. A–C. Per-subject Character (A), Word (B), and Semantic (C) Error Rate for three decoders: Encoder – MEG encoder greedy CTC predictions; Encoder + N-gram – encoder predictions decoded with a 6-gram character language model; Brain2Qwerty v2 – full pipeline conditioned on both CTC text and MEG-derived word embeddings. Each dot is one subject; the black bar is the cross-subject mean, printed above each column. D. Per-sentence word-edit count for the Best, Median, and Worst Subject, sorted ascending; colour-matched dashed lines mark each subject’s perfect-decoded boundary (WER=0, annotated as “% perfect”). E. Decoded sentences from Brain2Qwerty v2 for the Best, Median, and Worst Subjects on three example targets. F. Decoded sentences from the three decoders (Brain2Qwerty v2, Encoder + N-gram, Encoder) on three example targets from the median subject (S01). In A–B, ground-truth sentences are shown next to “Target”; word substitutions and insertions are highlighted in red and deletions are denoted by [-]. Statistical tests in B–D: two-sided paired Wilcoxon signed-rank across n=9 subjects. Brackets show three adjacent decoder comparisons (lower level) plus the outer Encoder vs. Brain2Qwerty v2 comparison (upper level). Significance levels: ∗p<0.05, ∗∗p<0.01, ∗∗∗p<0.001. All annotated comparisons reach p=0.0039 (∗⁣∗), the minimum achievable for n=9 paired observations.
• Figure 4: CTC Tokenizer and tuned LLM adaptation cut WER by 20% over baseline alignments. A. Per-subject word error rate for three contrastive token alignment strategies: Patch Tokenizer (fixed number of patches), Sentence Alignment (one embedding per sentence), CTC Tokenizer (CTC-segmented embeddings based on the predicted space token). B. Word-count calibration of the CTC Tokenizer. C. Example of word-level token retrieval using the CTC Tokenizer strategy. Each column lists the top-5 tokens corresponding to different words of the Qwen3-0.6B vocabulary ranked by cosine similarity to the brain-derived word embedding. A green cell marks the rank at which the ground-truth word is retrieved. D. LoRA rank sweep for Qwen3-0.6B (attention-only targets, α=2​r). Three training strategies are compared across LoRA ranks: All Subjects (single adapter for all subjects), Per-subject (independent adapters per subject), and Model Soup (uniform average of the per-subject model weights). Shaded bands show SEM across the 9 subjects, computed by subtracting each subject’s mean across conditions before taking the standard deviation so that they reflect within-subject variability only. E. LLM backbone scaling and LoRA target-module ablation. Each group shows two bars: All Subjects at r=2 (orange) and Model Soup at r=128 (green). The first three groups use attention-only LoRA with increasing backbone size (Qwen3-0.6B, 1.7B, 4B); the fourth uses Qwen3-4B with LoRA extended to all seven linear projections. Error bars use the same SEM definition than in D. All metrics are sentence-level WER, averaged per subject and then across the 9 subjects. Brackets report paired two-sided Wilcoxon signed-rank tests (p∗<0.05, p∗∗<0.01, p∗⁣∗∗<0.001); non-significant comparisons are unlabelled.
• Figure 5: Autonomous AI coding agents discover configurations that outperform classical optimization. A. Running-best validation WER on Subject S01 over successive experiments. Each dot represents one training run; colored step functions track the cumulative best WER for each method. The dashed grey line indicates the default configuration with only 4 parameters: learning rate, batch size, weight decay and rank with reasonable values. Optuna search is denoted with purple (dash-dotted). It searches within the 4-parameters space. Three independent AutoResearch agents are in blue, orange and green. B. Test WER evaluated on all 9 subjects for each method’s final configuration. Each dot represents one subject’s mean WER; colored horizontal lines indicate the cross-subject mean. Methods are ordered by decreasing (worse) WER from left to right. Significance brackets show paired Wilcoxon signed-rank tests between Optuna and each AutoResearch agent (∗∗∗ p<0.001).
• Figure S1: MEG sources over time relative to keystroke onset.
• Figure S2: tSNE clustering. A. Keyboard color map based on key location. B. tSNE clustering of key representations taken from the last layer of the Conformer of an untrained MEG Encoder, colored by subject (left) and key (right). C. tSNE clustering of key representations taken from the BrainModule (bottom row) and last layer of the Conformer (top row) of our trained MEG Encoder, colored by subject (left) and key (right).
• Figure S3: Decoded sentence examples across difficulty bands. 18 sentences decoded by Brain2Qwerty v2 for the Best, Median, and Worst subjects. Sentences are stratified by their mean Brain2Qwerty v2 WER, averaged across all repetitions across all 9 subjects, into three terciles. Six sentences randomly drawn from each: Best Decoded Sentences (bottom tercile, top row), Median Decoded Sentences (middle tercile, middle row), and Worst Decoded Sentences (top tercile, bottom row). Each prediction is rendered with per-word colour coding: correct words in black, substitutions / insertions in red, and deletions marked [-] in orange. Per-row WER is shown on the right.
• Figure S4: Brain2Qwerty v2 is a neuroLLM, not a corrector of CTC predictions. A–C. Per-subject Character (A), Word (B), and Semantic (C) Error Rate for two configurations of our LLM-based decoder that share the same Qwen3-0.6B backbone. Brain2Qwerty + LLM (grey) conditions the LLM only on the encoder’s predictions; Brain2Qwerty v2 (green) additionally conditions the LLM on the MEG-derived word embeddings (“Neuro Tokens”). Each pair of dots is one subject (n=9); thin black lines connect the same subject across the two configurations. Black horizontal bars are cross-subject means, printed beside each bar. Brackets: two-sided paired Wilcoxon signed-rank, ∗∗p<0.01.
• Figure S5: Encoder CER linearly predicts our model’s performance, and architecture choices set the encoder CER. Y-axis is shared by the three panels and represents the MEG Encoder character error rate (CER) on the test set. A. Per-sentence Brain2Qwerty v2 word error rate (WER) versus MEG Encoder CER, averaged across subjects per unique sentence. The green line is a linear regression. B. Same as (A) with semantic error rate (SemER) on the x-axis. C. Per-subject MEG Encoder CER for three encoder architectures: Temporal Patch Transformer, BrainModule Transformer, and the MEG Encoder used in this paper. All three pairwise comparisons are significant (paired Wilcoxon, two-sided).

最新论文

• SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?让AI编程助手去修复真实科学软件,连最强的那个也有一半以上任务没做对arXiv:2608.197992026-08-21
• FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving把稀疏注意力从论文原型变成能真正上线服务的加速方案arXiv:2608.197582026-08-21
• PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM Agents让客服AI坐席不只是拦住一个危险动作,而是把整个流程走对arXiv:2608.198612026-08-21
• EXIMO: VLM Guided Exploration of VLA Policies不用人工遥控演示,让会说话的AI来教机械臂做新家务arXiv:2608.198912026-08-21
• EnvHarness: Awakening Static Worlds for Agent Learning不重新搭建训练环境,而是给现有环境套一层可插拔组件,针对每个智能体的具体弱点重新塑形arXiv:2608.198802026-08-21
• Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the Model租用AI而非拥有AI的机构,安全监管能力只剩一半arXiv:2608.192162026-08-21
• Beyond Imitation: Filtering On-Policy Distillation by Reasoning ProgressAI模仿老师模型学习时,会误伤本来推理正确的步骤,新方法专门过滤掉这种误伤arXiv:2608.194082026-08-21
• PersonalBench: Measuring the Authorship Gap in LLM Personalization让AI模仿某人的文风,结果发现它始终摆脱不了自己的腔调arXiv:2608.197462026-08-21

METAL MEDIA 最新报道

• TUIDE出道曲《SUN KISS》在冰岛取景，打造K-pop公路电影
• NCT 127发行时隔两年全新正规专辑《BLINGY》