Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox

METAL MEDIA

Interpretable MEG Decoding of Perceived Speech: Cortical Sources and the Stimulus Features That Drive Retrieval

arXiv:2608.014812026-08-01

Cracking open an AI that guesses which 3-second speech clip you just heard from brain scans, to see exactly which brain regions and sound features it actually relies on

This work builds a deep learning model that retrieves brief speech clips a person heard, using only non-invasive brain recordings (MEG), while making its internal weights interpretable as real cortical locations and rhythms. On the MEG-MASC dataset with 27 participants, it picks the correct audio clip out of 1005 candidates with 39.75% Top-1 accuracy while using about 20 times fewer parameters than a prior comparable model. Mapping the learned weights back to the brain revealed sources consistent with the known speech-perception network, and a feature-removal experiment showed that 15 of 19 tested speech properties (like silence, loudness, vowels, and acoustic onsets) genuinely drive the decoding.

METAL MEDIA explanatory visual

Interpretable MEG-to-speech retrieval pipeline

Evidence statusMeasured results reported

  1. 1. 3D spatial attentionMaps 208 sensor signals to 270 virtual channels using spherical harmonics fitted to the sphere-like MEG helmet geometry
  2. 2. Interpretable branches (K=25)A participant-specific projection reduces the 270 channels to K=25 branches, each matched to a source via a 150 ms temporal filter
  3. 3. Non-linear decoderTwo residual convolutional blocks process the branch signals into a MEG embedding aligned with the wav2vec audio embedding
  4. 4. Source mappingTrained weights are projected into brain space via MNE and RAP-MUSIC, revealing auditory, frontal, and medial temporal lobe sources and left-right frequency differences
  5. 5. Paired occlusion testSwapping real MEG between feature-present and feature-absent intervals for 19 speech features confirms 15 of them drive retrieval
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

  1. Prior MEG-to-speech retrieval models (e.g., Défossez et al.) worked well but their learned weights couldn't be tied to real cortical locations or rhythms, making them black boxes. This work redesigns the spatial attention layer to use spherical harmonics, matching the roughly spherical shape of the MEG sensor helmet, instead of the earlier 2D Fourier functions built for a flattened sensor layout.
  2. The 270-channel participant-specific representation is compressed to just K=25 interpretable 'branches,' and each branch is given its own trainable 150 ms temporal filter, so every branch corresponds to a specific neural source in both space and time. Eye-movement and heartbeat artifacts are removed before training so the model can't shortcut its task using non-neural signals.
  3. Trained six times independently on the MEG-MASC dataset (27 participants), the model reaches 39.75±0.34% Top-1 accuracy and 70.4% Top-10 accuracy at picking the correct clip among 1005 candidates, using only 486,619 trainable decoder parameters versus about 9.57 million for the comparable Défossez et al. setup — roughly a 20x reduction.
  4. Projecting the trained weights into brain source space (using the method of Petrosyan et al.) recovered activity consistent with the canonical speech-perception network, including bilateral auditory cortex, frontal, and medial temporal lobe regions, with branches localized to the left hemisphere showing higher-frequency rhythmic components than those on the right.
  5. A paired-occlusion experiment swapped real MEG segments between intervals with and without each of 19 speech features (silence, loudness, vowels, acoustic onsets, etc.), finding that 15 of the 19 features significantly affected retrieval, with silence, loudness, vowels, and acoustic onsets having the largest effects — while randomly ordered word lists (lacking narrative structure) carried less recoverable information than coherent speech.
Figure 1: Interpretable front-end made as a collection of branches, the k-th branch is highlighted in blue. During training each branch gets matched to a neural source with specific spatial and dynamical properties.
Figure 1: Interpretable front-end made as a collection of branches, the k-th branch is highlighted in blue. During training each branch gets matched to a neural source with specific spatial and dynamical properties.
Figure 2: Our network’s architecture for MEG-to-audio embedding alignment. A 3-second, 208-channel MEG segment is processed by an interpretable front end: spherical-harmonic 3D spatial attention maps the sensor signals to 270 geometry-constrained virtual channels; a 1×1 unmixing convolution applies a learned affine transformation in this channel space; the subject-specific layer then projects the representation to K interpretable branches selected by the subject ID. Each branch is passed through a depthwise temporal convolution with a 150 ms kernel, producing filtered branch-wise signals. These signals are processed by a convolutional module with B residual convolutional blocks, where we evaluate B∈{0,…,5} and use B=2 in the main architecture, followed by a convolutional head that outputs the MEG embedding aligned with the wav2vec audio embedding.
Figure 2: Our network’s architecture for MEG-to-audio embedding alignment. A 3-second, 208-channel MEG segment is processed by an interpretable front end: spherical-harmonic 3D spatial attention maps the sensor signals to 270 geometry-constrained virtual channels; a 1×1 unmixing convolution applies a learned affine transformation in this channel space; the subject-specific layer then projects the representation to K interpretable branches selected by the subject ID. Each branch is passed through a depthwise temporal convolution with a 150 ms kernel, producing filtered branch-wise signals. These signals are processed by a convolutional module with B residual convolutional blocks, where we evaluate B∈{0,…,5} and use B=2 in the main architecture, followed by a convolutional head that outputs the MEG embedding aligned with the wav2vec audio embedding.
Table 1: Trainable model size and retrieval performance for representative configurations. F denotes the dimensionality of the target audio representation. The K=270, five-block LISA model is the closest tested configuration to Défossez et al. [15] in branch count and decoder depth, but is not a reproduction: it retains our 3D attention, temporal filters, preprocessing, and training procedure. The published Défossez et al. scores use a different preprocessing and evaluation protocol (word-aligned test segments and without ocular or cardiac component removal) and are included only to ground the external parameter comparison.
ModelKBFParametersCandidatesTop-1 (%)Top-10 (%)
LISA, smaller branch space152768380,769100539.5170.23
LISA, LinearDR-1225212427,651100539.9570.54
LISA, no convolutional blocks250768471,219100536.7667.37
LISA, main252768486,619100540.0170.60
LISA, five convolutional blocks255768509,719100539.0669.82
LISA, closest tested capacity27057687,210,224100536.4167.46
Défossez et al. [15]270510249,565,054136341.3070.70
Figure 3: Retrieval accuracy as a function of the number of interpretable branches K and the number of convolutional blocks in the decoder. For each configuration, Top-1 and Top-10 test accuracy are reported at the checkpoint with the lowest validation loss. Accuracy increases sharply from very small K to approximately K=10–25, then enters a broad plateau; larger K does not produce systematic gains and can mildly degrade performance. Across decoder depths, the 0-block model is consistently weaker, while models with 2–5 convolutional blocks form a similar high-performing regime. The main 2-conv, K=25 configuration lies on this compact high-accuracy plateau.
Figure 3: Retrieval accuracy as a function of the number of interpretable branches K and the number of convolutional blocks in the decoder. For each configuration, Top-1 and Top-10 test accuracy are reported at the checkpoint with the lowest validation loss. Accuracy increases sharply from very small K to approximately K=10–25, then enters a broad plateau; larger K does not produce systematic gains and can mildly degrade performance. Across decoder depths, the 0-block model is consistently weaker, while models with 2–5 convolutional blocks form a similar high-performing regime. The main 2-conv, K=25 configuration lies on this compact high-accuracy plateau.
Figure 4: 3D spherical-harmonic attention learned by architectures with varying numbers of non-linear convolutional blocks (B) and branch counts (K=5,10,25). Neff=(∑mpm2)−1 is the inverse-Simpson effective number of sensors, where pm=c¯m/∑m′c¯m′; smaller values indicate that attention is concentrated on fewer sensors. For visualization maximal value was capped to 99-th percentile, but all Neff are computed with full attention weights without clipping.
Figure 4: 3D spherical-harmonic attention learned by architectures with varying numbers of non-linear convolutional blocks (B) and branch counts (K=5,10,25). Neff=(∑mpm2)−1 is the inverse-Simpson effective number of sensors, where pm=c¯m/∑m′c¯m′; smaller values indicate that attention is concentrated on fewer sensors. For visualization maximal value was capped to 99-th percentile, but all Neff are computed with full attention weights without clipping.

Findings

  • Averaged over six independently trained models on MEG-MASC, the system reached 39.75±0.34% Top-1 and 70.4% Top-10 accuracy retrieving the correct clip among 1005 candidates, with only 486,619 decoder parameters versus about 9.57 million for a comparably-sized Défossez et al. setup, roughly a 20x reduction.
  • In a controlled comparison, the K=270, five-block configuration closest to the Défossez et al. design used 14.8 times more parameters yet scored 3.60 percentage points lower on Top-1 and 3.14 points lower on Top-10 accuracy.
  • Mapping trained weights to source space recovered activity in bilateral auditory cortex, frontal lobe, and medial temporal lobe, with left-localized branches showing higher-frequency rhythmic components than right-localized ones.
  • 15 of 19 tested stimulus features (including silence, loudness, vowels, and acoustic onsets) showed a statistically significant contribution to retrieval, and this pattern held across six independently trained model instances with different random seeds.
  • Reducing the target wav2vec representation to about 12 learned feature dimensions preserved retrieval accuracy, while a comparable reduction using fixed PCA degraded performance faster; compressing the temporal resolution of the target, by contrast, caused a clear performance drop under any tested method.
Figure 5: Leading ten singular vectors of the across-subject spatial filter and spatial pattern matrices aggregated from the interpretable branches of all 27 subjects, and the equivalent current dipoles fitted to the subspace spanned by these topographies. We can clearly observe the involvement of not only the primary auditory cortices but also frontal and medial temporal lobe structures.
Figure 5: Leading ten singular vectors of the across-subject spatial filter and spatial pattern matrices aggregated from the interpretable branches of all 27 subjects, and the equivalent current dipoles fitted to the subspace spanned by these topographies. We can clearly observe the involvement of not only the primary auditory cortices but also frontal and medial temporal lobe structures.
(b) Dominant dipoles derived from the cross-subject spatial patterns of Figure 5(a) using the RAP-MUSIC algorithm [51] (subspace correlation threshold 0.8). Dipoles cluster in bilateral auditory cortices, medial temporal lobe and frontal lobe on the fsaverage anatomy.
(b) Dominant dipoles derived from the cross-subject spatial patterns of Figure 5(a) using the RAP-MUSIC algorithm [51] (subspace correlation threshold 0.8). Dipoles cluster in bilateral auditory cortices, medial temporal lobe and frontal lobe on the fsaverage anatomy.

Where it can be used

  • Serving as a diagnostic tool to check which brain regions and signal properties a speech-decoding model genuinely relies on, useful when developing speech neuroprosthetics or imagined-speech interfaces
  • Assisting non-invasive language-area mapping efforts before surgery by cross-checking model-identified sources against known speech-perception regions
  • Providing a data-driven, hypothesis-free approach for neuroscience research into which acoustic and linguistic features are encoded in brain activity during natural speech perception
Figure 6: The 12 largest clusters for the main K=25 model, computed using the Cable Spool Fort recordings from the first session. Each column shows the medoid of one cluster. Rows show, from top to bottom, the sensor-space spatial pattern, the temporal pattern computed using the zero-mean temporal kernel, its magnitude spectrum, and the corresponding MNE-Python [30] source-magnitude estimate on the fsaverage surface in left- and right-hemisphere lateral views.
Figure 6: The 12 largest clusters for the main K=25 model, computed using the Cable Spool Fort recordings from the first session. Each column shows the medoid of one cluster. Rows show, from top to bottom, the sensor-space spatial pattern, the temporal pattern computed using the zero-mean temporal kernel, its magnitude spectrum, and the corresponding MNE-Python [30] source-magnitude estimate on the fsaverage surface in left- and right-hemisphere lateral views.
Figure 7: Paired MEG occlusion effects for 19 stimulus features. For each participant, the plotted effect is the retrieval-rank difference between feature-absent replacement (“removal”) and matched feature-present replacement (“control”), after averaging donor realisations, eligible queries, and multiple sessions. Positive values indicate that preserving feature-associated MEG information retained a better rank. Grey points show participant effects and diamonds show group means. Green violins show the feature-wise sign-flip null distributions in rank-difference units; stars mark one-sided single-step max-T familywise-error-corrected p<0.05. Every rank was computed against the complete, unchanged bank of 1005 candidates. Because the masks differed in duration and in their sets of eligible queries, effect magnitudes should not be read as a calibrated ranking of feature-encoding strength across features.
Figure 7: Paired MEG occlusion effects for 19 stimulus features. For each participant, the plotted effect is the retrieval-rank difference between feature-absent replacement (“removal”) and matched feature-present replacement (“control”), after averaging donor realisations, eligible queries, and multiple sessions. Positive values indicate that preserving feature-associated MEG information retained a better rank. Grey points show participant effects and diamonds show group means. Green violins show the feature-wise sign-flip null distributions in rank-difference units; stars mark one-sided single-step max-T familywise-error-corrected p<0.05. Every rank was computed against the complete, unchanged bank of 1005 candidates. Because the masks differed in duration and in their sets of eligible queries, effect magnitudes should not be read as a calibrated ranking of feature-encoding strength across features.

Limits and open work

  • The analysis relies on a single MEG corpus (MEG-MASC) with 27 participants, so it's untested whether the same feature-use pattern would replicate with different listening material or a different corpus.
  • Reproducibility across six differently-seeded models addresses dependence on model initialization only; robustness to a different architecture, dataset, or evaluation protocol has not been tested.
  • The front-end's interpretation assumes space-time factorized (linear) processing, which may fail to capture more complex, space-time inseparable dynamics such as propagating cortical waves.
  • The paired-occlusion swaps use real MEG from the same participant and session, and because features overlap, covary, and use masks of different durations, the method can't fully isolate each feature's independent causal contribution.
  • RAP-MUSIC source localization used a shared template forward model rather than individual anatomy for most participants, likely producing more spread-out cortical estimates than individualized models would; comparing the six participants with usable individual anatomy to this template result is left for future work.
Figure 8: Test retrieval accuracy as a function of paired MEG–audio segment duration for models with K=25 branches and B∈{0,2,5} convolutional blocks. Each point reports final-test accuracy from the checkpoint with the lowest validation loss. Audio embeddings were regenerated directly from the continuous sounds at every duration, and all conditions used the same 5 s-feasible anchor set and the same 991-candidate retrieval database. The 3 s points belong to this regenerated ablation and are distinct from the main 1005-candidate evaluation.
Figure 8: Test retrieval accuracy as a function of paired MEG–audio segment duration for models with K=25 branches and B∈{0,2,5} convolutional blocks. Each point reports final-test accuracy from the checkpoint with the lowest validation loss. Audio embeddings were regenerated directly from the continuous sounds at every duration, and all conditions used the same 5 s-feasible anchor set and the same 991-candidate retrieval database. The 3 s points belong to this regenerated ablation and are distinct from the main 1005-candidate evaluation.
Figure 9: Top-n test retrieval accuracy for the main two-block, K=25 model across paired MEG–audio segment durations. Curves were calculated using the similarity rank of the correct audio segment among the same 991 candidates. The advantage of longer segments is present throughout the evaluated range n=1,…,50.
Figure 9: Top-n test retrieval accuracy for the main two-block, K=25 model across paired MEG–audio segment durations. Curves were calculated using the similarity rank of the correct audio segment among the same 991 candidates. The advantage of longer segments is present throughout the evaluated range n=1,…,50.

Why it matters

Speech-decoding from brain signals underlies real applications like speech neuroprosthetics and mapping language areas during surgery, but high accuracy alone doesn't tell us what a black-box model actually detected or whether it's picking up genuine neural signal versus artifacts. By keeping accuracy competitive while making the model's weights traceable to specific brain sources, rhythms, and stimulus features, this work shows how a decoder can double as a scientific instrument for studying speech perception, not just a benchmark score.

Figure 10: Feature-space compression of the target wav2vec representation. The feature dimension is reduced using either a fixed PCA projection or a trainable linear projection optimized with the retrieval loss. Learned feature reduction preserves retrieval accuracy over a wide range of dimensions, whereas PCA degrades substantially faster in the low-dimensional regime.
Figure 10: Feature-space compression of the target wav2vec representation. The feature dimension is reduced using either a fixed PCA projection or a trainable linear projection optimized with the retrieval loss. Learned feature reduction preserves retrieval accuracy over a wide range of dimensions, whereas PCA degrades substantially faster in the low-dimensional regime.
Figure 11: Temporal-resolution compression of the target wav2vec trajectory. Temporal PCA, trainable linear reduction, local pooling, and global pooling methods are compared. Unlike feature compression, temporal compression causes a clear performance loss, and global pooling collapses to near-chance retrieval.
Figure 11: Temporal-resolution compression of the target wav2vec trajectory. Temporal PCA, trainable linear reduction, local pooling, and global pooling methods are compared. Unlike feature compression, temporal compression causes a clear performance loss, and global pooling collapses to near-chance retrieval.

Terms in this paper

  • MEG (magnetoencephalography) · a non-invasive brain imaging technique that measures the tiny magnetic fields produced by neural activity from outside the head
  • wav2vec 2.0 · a pretrained AI model that converts raw audio into vector representations (embeddings)
  • CLIP-style contrastive objective · a training method that pulls matching pairs of data from two different modalities (here, MEG and audio) into a shared vector space
  • spherical harmonics · mathematical basis functions defined on a sphere, a natural fit for the roughly spherical arrangement of MEG sensors
  • paired occlusion · an experiment that swaps MEG segments between feature-present and feature-absent intervals to test whether that feature affects the model's output

Original abstract (English)

Short segments of perceived speech can be retrieved from non-invasive magnetoencephalographic (MEG) recordings by deep networks trained with a CLIP-style objective against wav2vec 2.0 audio embeddings. Yet their weights do not map onto electrophysiological quantities, and it remains unclear which speech properties drive retrieval. We build on a high-performing MEG-to-audio retrieval architecture but redesign both its front end and decoder. Its spatial attention operates on a flattened sensor lay

Authors · Ilia Semenkov

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: Ilia Semenkov et al., arXiv:2608.01481, arxiv-nonexclusive