1 of 24

Multi-view and Cross-view Brain Decoding

Subba Reddy Oota*1,2, Jashn Arora*2, Manish Gupta2,3, Bapi Raju Surampudi2 

August 13, 2022

COLING 2022

1Inria Bordeaux France, 2IIIT-Hyderabad, 3Microsoft India

2 of 24

What is fMRI?

Coling-2022

2

https://www.biopac.com/events/fmri-psych/

A vision-language task in the scanner

Concept + Picture

fMRI Brain Activity

3 of 24

Encoding vs. Decoding

Coling-2022

3

Haiguang Wen et al,2017

Encoding

Decoding

Stimulus

Representation

Stimulus

Representation

fMRI

fMRI

4 of 24

What is Brain Decoding?

  • Can you read the mind with fMRI?
  • Or at least tell what the person saw?

Coling-2022

4

Visual Task

Language Task

Smith et al., 2011, Wang et al. 2019

5 of 24

Classical Decoders

  • Classical decoding solutions extracting linguistic meaning from imaging data have been largely limited to
    • concrete nouns,
    • using similar stimuli for training and testing,
    • small number of semantic categories.

Coling-2022

5

Mitchell et al. 2008

6 of 24

Multi-view and Cross-view Brain Decoding

  • Human brains have the unique capability of language acquisition:
    • the process of learning the language
    • understand the meaning of concepts from multiple modalities such as images, text, speech, and videos.
  • Prior works focus on single-view brain decoding using traditional feature engineering.
  • However, how the brain captures the meaning of linguistic stimuli across multiple views is still a critical open question in neuroscience.
  • Consider three different views of the concept bird:
    • (1) sentence using the target word,
    • (2) picture presented with the target word label, and
    • (3) word cloud containing the target word along with other semantically related words.
  • Earlier works have explored which of these three different views provides richer information to understand the concept.

​

Coling-2022

6

7 of 24

Dataset Details (Experiment-1)

Coling-2022

7

Concept + Sentence View

Concept Word

Concept + Picture View

Concept + Wordcloud View

Periera et al. 2018

8 of 24

Informative Voxel Selection

Coling-2022

8

Voxel + 26 neighbors in 3D

Input

Ridge Regression

Output

Stimulus:

Apartment

Present

BERT

Present

Stimulus:

Apartment

Pearson Correlation (R) = Corr(Y, W(X))

Correlation across feature dimensions

V1 – R1

V2 – R2

….

Vn – R3

Select 5000 voxels based on top-5000 correlation scores

3D Image

X

Y

W

9 of 24

Brain Decoder Schematic? (concept+picture)

Coling-2022

9

Present

Stimulus:

Apartment

Stimulus:

Apartment

Present

Ridge Regression

Periera et al. 2018, Devlin et al. 2018

BERT

BERT

10 of 24

Brain Decoder Schematic? (concept+sentence)

Coling-2022

10

Present

Stimulus:

Apartment

Stimulus:

Apartment

Present

Ridge Regression

Periera et al. 2018, Devlin et al. 2018

BERT

11 of 24

Evaluating Decoding Models: Pairwise Accuracy

Coling-2022

11

ith Concept Word

jth Concept Word

 

Periera et al. 2018

12 of 24

Evaluating Decoding Models: Rank Accuracy

Coling-2022

12

Y1

 

Y2

Yn

Periera et al. 2018

ith Concept Word

Correlation

 

rank = rsort(corr_scores).index(correlation)

All the correlation scores in descending order

13 of 24

Multi-view decoding

Coling-2022

13

Wordcloud View

Train

Sentence View

Picture View

Wordcloud View

Picture View

Train

Sentence View

Train

14 of 24

Multi-view decoding results

Coling-2022

14

Picture View

Train

BERT Representaions

Shuffled the Target Concepts

Test

Sentence View

Train

WordCloud View

Train

Pictures Best Accuracy

Sentences Best Accuracy

15 of 24

Distribution of Informative Voxels

Coling-2022

15

16 of 24

Distribution of Information Voxels: Language Network

Coling-2022

16

17 of 24

Distribution of Informative Voxels: Visual Network

Coling-2022

17

18 of 24

Cross-view Decoding

Coling-2022

18

Picture View

Train

Caption

Test

Picture View

Train

Visual words

Test

Wordcloud View

Train

Sentence

Test

Sentence View

Train

Keywords

Test

19 of 24

Cross-view Decoding results

Coling-2022

19

BERT Representaions

Shuffled the Target Concepts

20 of 24

Distribution of Informative Voxels

Coling-2022

20

21 of 24

Distribution of Informative Voxels: Language Network

Coling-2022

21

22 of 24

Distribution of Information Voxels: Visual Network

Coling-2022

22

23 of 24

Insights

  • Without any apriori location constraints,
    • nearly half of the voxels selected by the models fall into the language atlas.
    • Conclusion: brain regions active in language processing also highly correspond to these representations.

Coling-2022

23

24 of 24

Collaborators

Subba Reddy Oota

Jashn Arora

Manish Gupta

Bapi Raju Surampudi