1 of 24

Speech Taskonomy: Which Speech Tasks are the most Predictive of fMRI Brain Activity?

1

Subba Reddy Oota¹

Veeral Agarwal²

Mounika Marreddy²

Manish Gupta^2,3

Bapi Raju Surampudi²

¹Inria Bordeaux, France ²IIIT-Hyderabad, India ³Microsoft, India

2 of 24

What is fMRI?

2

A listening task in the scanner

Narrative Story

fMRI Brain Activity

https://www.biopac.com/events/fmri-psych/

3 of 24

Brain Encoding vs Decoding

3

Encoding

Stimulus

Representation

fMRI

Yizhen Zhang, Kuan Han, Robert Worth & Zhongming Liu. "Connecting concepts in the brain by mapping cortical representations of semantic relations" Nature (2020).

Stimulus

Representation

Decoding

fMRI

4 of 24

4

Method to study alignment of SM & brain representations

Brain alignment of a SM ⇒ how similar its representations are to a human brain’s

Speech Model (SM)

Toneva, M., & Wehbe, L. (2019). Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). Advances in Neural Information Processing Systems, 32.

We follow a general method used in prior works to study the alignment of representations between a LM and a human brain.

[CLICK] We provide the same chapter from a Harry Potter book to both the Language model as well as human participants.

[CLICK] For the Language model, we extract its representations at every layer for every word in the text.
[CLICK] For the human participants, we record their brain activity using Functional magnetic resonance imaging (or fMRI)
[CLICK] Next, we train a linear function that uses the extracted Language model representations to predict human brain activity
[CLICK] Finally, we test this function on unseen data, and evaluate its accuracy as the amount of “brain alignment”

[CLICK]

In summary, brain alignment of a Language model refers to how similar its representations are to a human brain’s.

5 of 24

Brain Encoding?

Present

Stimulus

Ridge Regression

Input

Output

X

Y

W

Pearson Correlation (R) = Corr(Y, W(X))

6 of 24

Encoding: training independent models

Independent model per participant

Independent model per voxel / sensor-timepoint

P1

…

P2

PN

P1, v1

P1, v2

…

P1, vm

7 of 24

Recent work utilizing progress in self-supervised speech models for encoding

Vaidya, Aditya R., Shailee Jain, and Alexander G. Huth. "Self-supervised models of audio effectively explain human cortical responses to speech." ICML (2022).

Stimuli: Moth Radio Hour
Stimulus representation: derived from pretrained self-supervised speech models (HuBERT, Wav2Vec2.0, APC)
Brain recording & modality: fMRI, listening

Middle layers of self-supervised speech models predict auditory cortex the best

8 of 24

Audio work utilizing DL progress

Millet, Juliette, Charlotte Caucheteux, Pierre Orhan, Yves Boubenec, Alexandre Gramfort, Ewan Dunbar, Christophe Pallier, and Jean-Remi King. "Toward a realistic model of speech processing in the brain with self-supervised learning." arXiv preprint arXiv:2206.01685 (2022).

Stimuli: audio books
Stimulus representation: derived from pretrained self-supervised speech model (Wav2Vec2.0)
Brain recording & modality: fMRI, listening in 3 languages (Eng, Fr, Mandarin)

Self-supervised speech models exhibit specialization for native sounds in the STS and MTG.
IFG and AG show more general specialization for speech rather than native-language

Does the model show commonalities merely with general auditory processing in the brain, or does it capture speech-specific processing? If so, does it show commonalities with brain representations that are specific to the native language of the participants, or merely to general speech processing? We first evaluate the specialization of humans’ perception to their native language using an ABX behavioral task (Section 2.4). Specifically, we compare 386 French and English participants on their ability to distinguish native and non-native phones. As expected Bohn [2017], Kuhl et al. [2005], participants were better at discriminating native sounds than non-native ones (across phone pairs: p < 10−18, Figure 4-A). Second, applying the same test to our self-supervised French and English models shows that, like humans, models best discriminate sounds from their ‘native’ language (i.e., the French model better distinguishes French stimuli than English ones, across phone pairs, and vice versa: p < 0.05). Finally, the random and acoustic models obtain the worst ABX accuracy. These results confirm that 600 h of self-supervised learning suffices for Wav2Vec 2.0 to learn language-specific representations (Figure 4-B).

Wav2Vec 2.0 and the brain learn language specific representations. Next, we compare the Neural Predictivity scores of random, non-speech, non-native and native models (Figure 4-C D). First, our results show that the non-speech model attains higher R scores than the random model (on average across voxels, ∆R = 0.006, p=10−31) confirming the importance of learning to generate brain-like representations. Second, non-native models attain higher R scores than the non-speech model (∆R = 0.002, p = 10−9 ), confirming that Wav2Vec 2.0 learns speech-specific representations of sounds when trained on speech. Finally, the native model attains higher R scores than non-native models (∆R = 0.002, p = 10−15). The specialization for native sounds shared between models and the brain appears to be mainly localized around the superior temporal sulcus and the middle temporal gyrus (Figure 4-D), whereas higher-level regions like IFG and the angular gyrus only show more general specializations (i.e., speech rather than native-language sounds).

9 of 24

Audio work utilizing DL progress

Subba Reddy Oota, Khushbu Pahwa, Mounika Marreddy, Manish Gupta, and Bapi S. Raju. "Neural architecture of speech" ICASSP-2023

Stimuli: Moth-Radio-Hour
Stimulus representation: derived from 5 basic + 25 pretrained self-supervised speech models
Brain recording & modality: fMRI

Contrastive and predictive models encode the information better than the generative and the traditional low-level acoustic baselines, and VGGish models.

Data2Vec

Does the model show commonalities merely with general auditory processing in the brain, or does it capture speech-specific processing? If so, does it show commonalities with brain representations that are specific to the native language of the participants, or merely to general speech processing? We first evaluate the specialization of humans’ perception to their native language using an ABX behavioral task (Section 2.4). Specifically, we compare 386 French and English participants on their ability to distinguish native and non-native phones. As expected Bohn [2017], Kuhl et al. [2005], participants were better at discriminating native sounds than non-native ones (across phone pairs: p < 10−18, Figure 4-A). Second, applying the same test to our self-supervised French and English models shows that, like humans, models best discriminate sounds from their ‘native’ language (i.e., the French model better distinguishes French stimuli than English ones, across phone pairs, and vice versa: p < 0.05). Finally, the random and acoustic models obtain the worst ABX accuracy. These results confirm that 600 h of self-supervised learning suffices for Wav2Vec 2.0 to learn language-specific representations (Figure 4-B).

Wav2Vec 2.0 and the brain learn language specific representations. Next, we compare the Neural Predictivity scores of random, non-speech, non-native and native models (Figure 4-C D). First, our results show that the non-speech model attains higher R scores than the random model (on average across voxels, ∆R = 0.006, p=10−31) confirming the importance of learning to generate brain-like representations. Second, non-native models attain higher R scores than the non-speech model (∆R = 0.002, p = 10−9 ), confirming that Wav2Vec 2.0 learns speech-specific representations of sounds when trained on speech. Finally, the native model attains higher R scores than non-native models (∆R = 0.002, p = 10−15). The specialization for native sounds shared between models and the brain appears to be mainly localized around the superior temporal sulcus and the middle temporal gyrus (Figure 4-D), whereas higher-level regions like IFG and the angular gyrus only show more general specializations (i.e., speech rather than native-language sounds).

10 of 24

Challenges in using DL for cognitive science

Not designed to specifically model brain processing

Training DL models using brain recordings
Task-based modeling

Can be difficult to interpret due to multiple sources of information

Disentangling contributions of different info sources to brain predictions

10

11 of 24

Tasks affect processing

Wang, Aria, Michael Tarr, and Leila Wehbe. "Neural taskonomy: Inferring the similarity of task-derived representations from brain activity." Advances in Neural Information Processing Systems 32 (2019).

11

Stimuli: images of natural scenes
Stimulus representation: task-optimized CNNs for a range of tasks
Brain recording & modality: fMRI, vision

Semantic Low-dim. Geometric 2D 3D

Vision tasks with higher transferability make similar predictions for brain responses from different regions

Convolutional neural networks (CNNs) trained for object classification have been widely used to account for visually-driven neural responses in both human and primate brains. However, because of the generality and complexity of object classification, despite the effectiveness of CNNs in predicting brain activity, it is difficult to draw specific inferences about neural information processing using CNNderived representations. To address this problem, we used learned representations drawn from 21 computer vision tasks to construct encoding models for predicting brain responses

Features from 3D tasks such as keypoint/edge detection explain greater variance compared to 2D tasks—a pattern observed across the whole brain. Using results across all 21 task representations, we constructed a “task graph” based on the spatial layout of well-predicted brain areas from each task. A comparison of this brain-derived task structure to the task structure derived from transfer learning accuracy demonstrate that tasks with higher transferability make similar predictions for brain responses from different regions.

In the Taskonomy result, tasks are clustered into 3D (indicated in green), 2D (blue), low-dimensional geometric (red) and semantic (purple) tasks.

Interestingly, the tree derived from brain representation also shows a similar structure: semantic, 2D and 3D tasks are clustered together.

12 of 24

Tasks affect processing

Oota, Subba Reddy, Jashn Arora, Veeral Agarwal, Mounika Marreddy, Manish Gupta, and Bapi Raju Surampudi. "Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity?." arXiv preprint arXiv:2205.01404 (2022).

12

Stimuli: passages and narratives
Stimulus representation: task-optimized NLP models for a range of tasks
Brain recording & modality: fMRI, reading & listening of different stimuli

Reading fMRI best explained by coref. resolution, NER, shallow syntax parsing
Listening fMRI best explained by paraphrasing, summarization, NLI

13 of 24

Can task-specific speech models better predict fMRI brain activity?

Model fine-tuned to�downstream speech tasks

SUPERB (Speech Processing Universal PERformance Benchmark)

Ashish Seth, Vasista Sai Lodagala, Sreyan Ghosh, and S. Umesh. "ANALYZING THE FACTORS AFFECTING USEFULNESS OF SELF-SUPERVISED" https://arxiv.org/pdf/2203.16973.pdf

https://superbbenchmark.org

PR

SID

KS

SV

ASR

SD

ER

IC

14 of 24

14

Pretrained�speech model

Model fine-tuned to�downstream speech tasks

input

activations

Wav2Vec 2.0-base

15 of 24

15

Pretrained�speech model

Model fine-tuned to�downstream speech tasks

input

Wav2Vec 2.0-base

Use model’s internal layer�activations to predict brain activity on held-out data

activations

Compare against actual brain recordings

(brain alignment)

16 of 24

16

Pretrained�speech model

Model fine-tuned to�downstream speech tasks

input

Wav2Vec2.0-base

Use model’s internal layer�activations to predict brain activity on held-out data

activations

Compare against actual brain recordings

(brain alignment)

Hypothesis: If the pretrained model fine-tuned to downstream speech task has greater brain alignment than pretrained model, the downstream task is capturing more brain-relevant information

17 of 24

Listening data target: human brain recordings

17

We use Pieman story listening:

82 subjects,
282 TRs (repetition time)
here it is 1.5 sec.

Example: ''I began my illustrious carrier in journalism…''

Samuel A. Nastase. The “Narratives” fMRI dataset for evaluating models of naturalistic language comprehension. Nature, 2021.

18 of 24

18

Brain alignment – 4-fold Cross-Validation + Ridge regression

4-fold Cross-Validation

Linear regression regularized�with ridge penalty

Input: R^{282 x d}

d = embedding size, e.g. 4608

Prediction: R^{n x v}

n = number of fMRI intervals

v = number of voxels in participant’s brain

19 of 24

19

Brain alignment – Methodology

300 TRs x 768 ⇒ SM representations

Wav2Vec 2.0-base
SUPERB Benchmark downstream tasks : eight tasks

282 TRs x 768

Remove 18 TRs ⇒ 10 TRs in the beginning and 8 TRs in the ending (silent music)

282 fMRI time intervals x 4608

Concatenate SM representations for previous 6 TRs ⇒ fMRI response from brain activity peaks about 8-10 seconds after stimulus onset

282 fMRI intervals x 4608

Ridge regression (RR) ⇒ for each voxel, 282 data rows of 4608 parameters to predict 1 output
4-fold Cross-Validation to improve reliability

282 fMRI intervals x 52400 voxels ⇒ fMRI predictions (same dimensions as actual brain activity)

20 of 24

Results: Whole-Brain

20

ASR best encodes speech stimuli for brain response prediction.

ASR task has the best brain alignment in the middle layers.

Certain speech tasks (ASR, ER, SID and IC) that are important for improved brain alignment over pretrained Wav2Vec2.0.
SD and SV are not important in listening to stories.

21 of 24

Region level alignments

All speech tasks are better aligned with EAC compared to AAC and IFG regions.
Finetuning on ER, SID and IC leads to the best alignment for the early auditory cortex
Finetuning on ASR provides the best encoding for the auditory associative cortex and language regions.

Speech Taskonomy: Which Speech Tasks are the most Predictive of fMRI Brain Activity? Subba Reddy Oota, Veeral Agarwal, Mounika Marreddy, Manish Gupta, Raju Bapi. InterSpeech 2023

21

22 of 24

Sub-region level alignments

EAC: A1 has a higher Pearson correlation than other sub-ROIs.
Language ROIs 44 and 45, together with STSda and STSdp in AAC, are part of the well-known language network associated with narrative comprehension
ASR finetuned model performs best in these regions.

22

23 of 24

Limitations

We leveraged models finetuned using datasets of different sizes across tasks.
While a fair comparison of dataset sizes across tasks is impossible,

we understand that this could have resulted in some bias in our results.

24 of 24

Thank You

Questions?