Accurate Decoding of Natural Sentences from Non-Invasive Brain Recordings

What they did

1. Built Brain2Qwerty v2, which decodes full sentences from continuous, non-invasive MEG brain signals without needing surgery or precise keystroke timing.
2. Used a three-part pipeline: a character-level encoder trained with CTC, a word-level aligner using contrastive learning, and a fine-tuned large language model (LLM) that generates the final sentence.
3. Collected a much larger dataset than prior work (22,000 sentences, 10 hours per subject, nine subjects), and showed decoding accuracy improves log-linearly as more data is added.
4. Deployed autonomous AI coding agents to optimize the pipeline; they found configurations that beat a standard hyperparameter search (Optuna) and generalized across all subjects.
5. Achieved an average word error rate of 39% (22% for the best subject), roughly double the accuracy of the team's own prior non-invasive system.

Why it matters

This points toward restoring communication for people who have lost the ability to speak or move, without the medical risks of surgical brain implants. It also shows that simply collecting more data can help close the performance gap between non-invasive and invasive brain-computer interfaces.

Terms in this paper

• MEG (magnetoencephalography) · a non-invasive scanner that measures tiny magnetic fields from brain activity through the scalp
• CER/WER/SemER · error rates measuring how much a decoded sentence differs from the true one, at the character, word, or meaning level
• CTC (Connectionist Temporal Classification) · a training method that lets a model output a sequence of characters from continuous signals without knowing exact timing
• LoRA · a technique that fine-tunes a large language model by training only small added parameters instead of the whole model
• Model Soup · averaging the weights of several separately trained models into one combined model

Figures we cannot republish

• Figure 1: Asynchronous MEG decoding is unlocked by recording scale and variety. A. Experimental protocol. Left. We recorded healthy volunteers for 10 hours each using Magnetoencephalography (MEG) while they typed natural sentences they heard a few seconds prior. Right. Average MEG source reconstruction at the time of key press suggest that MEG primarily picks neural activity in the motor cortex. B. Approaches for brain-to-text decoding. Synchronous decoding consists in classifying the character from windows time-locked to each keystroke (e.g. levy2025brain). Asynchronous decoding consists in decoding text from a continuous brain signal, and can thus be applied in real-time, although with some potential delays (e.g. feghhi2025time). C. Quantity (hours per participant) and diversity (number of unique sentences) of our dataset (EnglishBCBL) as compared to levy2025brain (SpanishBCBL). D. Character-error-rate (CER) for the synchronous encoder of levy2025brain. Each coloured dot is one subject; the bar plots the across-subject mean. E. Same as D for our asynchronous encoder. F. Scaling of the asynchronous encoder CER as a function of the amount of training data (log scale), expressed as total recording hours per subject (test set fixed across all points). The blue curve is the across-subject mean on our EnglishBCBL dataset. The orange diamond places SpanishBCBL on the same axis after training our asynchronous encoder on it. G. Impact of sentence-list variety on asynchronous encoder CER at matched total sentence count: 128 unique sentences × 2 repetitions (SpanishBCBL protocol, orange) versus 256 unique sentences (EnglishBCBL protocol, blue). Two-sided Mann-Whitney U test across the n=9 subjects per condition: (***) denotes p < 0.001. Across all panels, CER is computed per sentence, then averaged within each subject, and finally averaged across subjects.
• Figure 2: Brain2Qwerty v2 architecture. Our pipeline is solely input with the continuous MEG recording corresponding to an entire typed sentence and outputs the decoded sentence thanks to three jointly-optimized modules. First, the Encoder is trained with a CTC loss (graves2006connectionist) to extract character-level representations from brain activity, and outputs both MEG Embeddings and a sequence of characters. Second, the Aligner learns, with a SigLIP loss (zhai2023sigmoid), to group and align the MEG embeddings with the true Word Embeddings. Finally, a Large Language Model (LLM) is input with both the MEG tokens and the Encoder’s text to generate the correct sentence autoregressively and with LoRA finetuning (huLoRALowRankAdaptation2021).
• Figure 3: Brain2Qwerty v2 enables word- and meaning-level decoding from MEG. A–C. Per-subject Character (A), Word (B), and Semantic (C) Error Rate for three decoders: Encoder – MEG encoder greedy CTC predictions; Encoder + N-gram – encoder predictions decoded with a 6-gram character language model; Brain2Qwerty v2 – full pipeline conditioned on both CTC text and MEG-derived word embeddings. Each dot is one subject; the black bar is the cross-subject mean, printed above each column. D. Per-sentence word-edit count for the Best, Median, and Worst Subject, sorted ascending; colour-matched dashed lines mark each subject’s perfect-decoded boundary (WER=0, annotated as “% perfect”). E. Decoded sentences from Brain2Qwerty v2 for the Best, Median, and Worst Subjects on three example targets. F. Decoded sentences from the three decoders (Brain2Qwerty v2, Encoder + N-gram, Encoder) on three example targets from the median subject (S01). In A–B, ground-truth sentences are shown next to “Target”; word substitutions and insertions are highlighted in red and deletions are denoted by [-]. Statistical tests in B–D: two-sided paired Wilcoxon signed-rank across n=9 subjects. Brackets show three adjacent decoder comparisons (lower level) plus the outer Encoder vs. Brain2Qwerty v2 comparison (upper level). Significance levels: ∗p<0.05, ∗∗p<0.01, ∗∗∗p<0.001. All annotated comparisons reach p=0.0039 (∗⁣∗), the minimum achievable for n=9 paired observations.
• Figure 4: CTC Tokenizer and tuned LLM adaptation cut WER by 20% over baseline alignments. A. Per-subject word error rate for three contrastive token alignment strategies: Patch Tokenizer (fixed number of patches), Sentence Alignment (one embedding per sentence), CTC Tokenizer (CTC-segmented embeddings based on the predicted space token). B. Word-count calibration of the CTC Tokenizer. C. Example of word-level token retrieval using the CTC Tokenizer strategy. Each column lists the top-5 tokens corresponding to different words of the Qwen3-0.6B vocabulary ranked by cosine similarity to the brain-derived word embedding. A green cell marks the rank at which the ground-truth word is retrieved. D. LoRA rank sweep for Qwen3-0.6B (attention-only targets, α=2​r). Three training strategies are compared across LoRA ranks: All Subjects (single adapter for all subjects), Per-subject (independent adapters per subject), and Model Soup (uniform average of the per-subject model weights). Shaded bands show SEM across the 9 subjects, computed by subtracting each subject’s mean across conditions before taking the standard deviation so that they reflect within-subject variability only. E. LLM backbone scaling and LoRA target-module ablation. Each group shows two bars: All Subjects at r=2 (orange) and Model Soup at r=128 (green). The first three groups use attention-only LoRA with increasing backbone size (Qwen3-0.6B, 1.7B, 4B); the fourth uses Qwen3-4B with LoRA extended to all seven linear projections. Error bars use the same SEM definition than in D. All metrics are sentence-level WER, averaged per subject and then across the 9 subjects. Brackets report paired two-sided Wilcoxon signed-rank tests (p∗<0.05, p∗∗<0.01, p∗⁣∗∗<0.001); non-significant comparisons are unlabelled.
• Figure 5: Autonomous AI coding agents discover configurations that outperform classical optimization. A. Running-best validation WER on Subject S01 over successive experiments. Each dot represents one training run; colored step functions track the cumulative best WER for each method. The dashed grey line indicates the default configuration with only 4 parameters: learning rate, batch size, weight decay and rank with reasonable values. Optuna search is denoted with purple (dash-dotted). It searches within the 4-parameters space. Three independent AutoResearch agents are in blue, orange and green. B. Test WER evaluated on all 9 subjects for each method’s final configuration. Each dot represents one subject’s mean WER; colored horizontal lines indicate the cross-subject mean. Methods are ordered by decreasing (worse) WER from left to right. Significance brackets show paired Wilcoxon signed-rank tests between Optuna and each AutoResearch agent (∗∗∗ p<0.001).
• Figure S1: MEG sources over time relative to keystroke onset.
• Figure S2: tSNE clustering. A. Keyboard color map based on key location. B. tSNE clustering of key representations taken from the last layer of the Conformer of an untrained MEG Encoder, colored by subject (left) and key (right). C. tSNE clustering of key representations taken from the BrainModule (bottom row) and last layer of the Conformer (top row) of our trained MEG Encoder, colored by subject (left) and key (right).
• Figure S3: Decoded sentence examples across difficulty bands. 18 sentences decoded by Brain2Qwerty v2 for the Best, Median, and Worst subjects. Sentences are stratified by their mean Brain2Qwerty v2 WER, averaged across all repetitions across all 9 subjects, into three terciles. Six sentences randomly drawn from each: Best Decoded Sentences (bottom tercile, top row), Median Decoded Sentences (middle tercile, middle row), and Worst Decoded Sentences (top tercile, bottom row). Each prediction is rendered with per-word colour coding: correct words in black, substitutions / insertions in red, and deletions marked [-] in orange. Per-row WER is shown on the right.
• Figure S4: Brain2Qwerty v2 is a neuroLLM, not a corrector of CTC predictions. A–C. Per-subject Character (A), Word (B), and Semantic (C) Error Rate for two configurations of our LLM-based decoder that share the same Qwen3-0.6B backbone. Brain2Qwerty + LLM (grey) conditions the LLM only on the encoder’s predictions; Brain2Qwerty v2 (green) additionally conditions the LLM on the MEG-derived word embeddings (“Neuro Tokens”). Each pair of dots is one subject (n=9); thin black lines connect the same subject across the two configurations. Black horizontal bars are cross-subject means, printed beside each bar. Brackets: two-sided paired Wilcoxon signed-rank, ∗∗p<0.01.
• Figure S5: Encoder CER linearly predicts our model’s performance, and architecture choices set the encoder CER. Y-axis is shared by the three panels and represents the MEG Encoder character error rate (CER) on the test set. A. Per-sentence Brain2Qwerty v2 word error rate (WER) versus MEG Encoder CER, averaged across subjects per unique sentence. The green line is a linear regression. B. Same as (A) with semantic error rate (SemER) on the x-axis. C. Per-subject MEG Encoder CER for three encoder architectures: Temporal Patch Transformer, BrainModule Transformer, and the MEG Encoder used in this paper. All three pairwise comparisons are significant (paired Wilcoxon, two-sided).

Latest papers

• SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the timearXiv:2608.197992026-08-21
• FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papersarXiv:2608.197582026-08-21
• PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad actionarXiv:2608.198612026-08-21
• EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise itarXiv:2608.198912026-08-21
• EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknessesarXiv:2608.198802026-08-21
• Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversightarXiv:2608.192162026-08-21
• Beyond Imitation: Filtering On-Policy Distillation by Reasoning ProgressA fix for AI models that get penalized by their teacher even when they're reasoning correctlyarXiv:2608.194082026-08-21
• PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappearsarXiv:2608.197462026-08-21

Latest from METAL MEDIA

• TUIDE debut with 'SUN KISS,' an Iceland-shot K-pop road movie
• NCT 127 drop BLINGY, their first full album in two years