Paper Walkthrough: Interpretable MEG Decoding of Perceived Speech — Reading the Decoder's Weights as a Brain Map

From three seconds of magnetoencephalography, this decoder picks the audio a listener was hearing out of 1005 candidates with 39.75% Top-1 accuracy — and, unlike its predecessors, its trained weights translate into cortical locations and rhythms, while paired occlusion experiments reveal which speech features it actually relies on.

Interpretable MEG Decoding of Perceived Speech: Cortical Sources and the Stimulus Features That Drive Retrieval

Guessing the voice in someone's head — from outside it

While a person listens to an audiobook, sensors placed around their head record the brain's magnetic field. Given just three seconds of that recording, the task is to pick which audio clip the person was hearing out of 1005 candidates. That is the problem this paper tackles, using magnetoencephalography (MEG) — a measurement that requires no surgery and no implanted electrodes. The model gets the right answer on the first guess about 39.75% of the time (§4.1). Random guessing would land at 0.1%, so that is roughly a 400-fold improvement over chance.

Here's the thing: the "guessing" part was already solved. Défossez et al. demonstrated this kind of retrieval in 2023 (§1). What this paper adds is not accuracy — it is the ability to read the reasons off the trained model.

An analogy: two detectives

Imagine two skilled detectives. The first has an excellent hit rate but never explains a conclusion. The second is just as accurate and can point to the evidence: "this footprint, plus that alibi." Only the second one is useful as an instrument of science.

Conventional deep decoders are the first detective. As the paper puts it, their weights match nothing an electrophysiologist can name — "not a cortical location, not a rhythm, not a time course" (§1). This work builds the second detective by constraining the network's front end with the physics of the measurement and the physiology of its sources, so that after training, the weights can be converted into "which part of the cortex, oscillating at what frequency, this branch was listening to."

Background: what does MEG actually measure?

The starting point is the physics of how MEG signals arise (§2.1). When many neurons in a cortical patch fire in synchrony, their combined current can be approximated by a single equivalent current dipole — think of it as a tiny battery — whose magnetic field reaches the M sensors surrounding the head. The signal $\mathbf{x}(t)$ measured at time $t$ is a sum:

Put in words: what the sensors pick up is the brain activity you want, plus brain activity that has nothing to do with the task, plus noise from the machine — all summed together. Only that sum is observable, and nothing in the recording carries a label saying which part came from where.

The first term is the task-related brain activity: $s_n(t)$ is the activation of the $n$-th neuronal source (a single waveform over time), and $\mathbf{g}_n$ is a vector describing how strongly that source reaches each of the M sensors — its topography, effectively the source's fingerprint. The second term is brain activity unrelated to the task, and $\mathbf{n}(t)$ is sensor noise. An MEG recording is like a stereo mix of many instruments playing at once; decoding means isolating the instruments you care about (§2.2).

Framing the task: retrieval, not reconstruction

Rather than reconstructing the waveform, this paper (following its predecessor) frames the problem as retrieval (§3). The audio side is encoded into embedding vectors by wav2vec 2.0, a self-supervised speech model; the MEG side is encoded into the same space by a neural network. Training uses a CLIP-style contrastive objective — pull matching pairs together, push mismatched pairs apart. At test time, the MEG embedding is compared against all 1005 candidate audio embeddings by cosine similarity, and the closest one wins.

The data is the public MEG-MASC dataset: 208-channel MEG from 27 English speakers listening to narrated stories. Audio is cut into 3-second windows with a 1-second stride, and each is paired with the MEG segment starting 150 ms later to account for auditory-response latency (§3.7). The test set is 1005 candidates built from the final seven pieces of the story Black Willow.

The design: constraining the network with physics and physiology

The paper starts from the Défossez et al. architecture and makes four changes (§1, §3). The guiding motive is not to raise the score but to make the model readable — though, as it turns out, the score does not suffer either.