Speech Taskonomy: Which Speech Tasks are the most Predictive of fMRI Brain Activity?
1
Subba Reddy Oota1,2
Veeral Agarwal2
Mounika Marreddy2
Manish Gupta2,3
Bapi Raju Surampudi2
1Inria Bordeaux, France 2IIIT-Hyderabad, India 3Microsoft, India
What is brain encoding?
Speech Taskonomy: Which Speech Tasks are the most Predictive of fMRI Brain Activity? Subba Reddy Oota, Veeral Agarwal, Mounika Marreddy, Manish Gupta, Raju Bapi. InterSpeech 2023
2
Encode
Speech
Deep
Learning
Models
Predict
Speech tasks from SUPERB benchmark.
Speech Taskonomy: Which Speech Tasks are the most Predictive of fMRI Brain Activity? Subba Reddy Oota, Veeral Agarwal, Mounika Marreddy, Manish Gupta, Raju Bapi. InterSpeech 2023
3
Experiments and Results
Speech Taskonomy: Which Speech Tasks are the most Predictive of fMRI Brain Activity? Subba Reddy Oota, Veeral Agarwal, Mounika Marreddy, Manish Gupta, Raju Bapi. InterSpeech 2023
4
ASR best encodes speech stimuli for brain response prediction.
ASR task has the best brain alignment in the middle layers.
Region and Sub-region level alignments
Speech Taskonomy: Which Speech Tasks are the most Predictive of fMRI Brain Activity? Subba Reddy Oota, Veeral Agarwal, Mounika Marreddy, Manish Gupta, Raju Bapi. InterSpeech 2023
5
Conclusion
Speech Taskonomy: Which Speech Tasks are the most Predictive of fMRI Brain Activity? Subba Reddy Oota, Veeral Agarwal, Mounika Marreddy, Manish Gupta, Raju Bapi. InterSpeech 2023
6