1 of 57

Insights from Neuroscience research

2 of 57

Fundamental questions about what information is processed where, when, and how in the brain

Where

When

What

How?

3 of 57

Where: set of left-lateralized regions more active for sentences than nonword lists

[Fedorenko et al. 2010, 2014]

Language Network

reading and listening naturalistic stories leads to bilateral activation of Language Network

[Wehbe et al. 2014, Huth et al. 2016]

4 of 57

When: word meaning processed 200-600ms post word onset

[Salmelin, 2007, Skeide and Friederici, 2016]

figure adapted from Salmelin, 2007

5 of 57

What: representation of word meaning distributed across cortex

[Barsalou, 1999; Barsalou, 2008; Pecher et al., 2005]

figure from Kemmerer, 2014; adapted from Thompson-Schill et al. 2006

Evidence for this organization for attributes related to:

audition [Kiefer et al., 2008]
color [Simmons et al., 2007]
shape [Chao et al., 1999]
motion [Damasio et al., 1996]
olfaction and taste [Goldberg, Perfetti, et al., 2006a; Goldberg, Perfetti, et al., 2006b]

6 of 57

The trophy doesn't fit into the brown suitcase because it's too big.

...

?

How

?

...

How

7 of 57

Typical studies of language processing with controlled experiments

7

How the human brain computes and encodes syntactic structures?

Syntax: how do words structurally combine to form sentences and meaning?

Controlled experiments

I believe that you should accept the proposal of your new associate

thing very tree where of watching copy tensed they states heart plus

Structured complex sentence

Word lists

Pallier et al. 2011

8 of 57

Language organization in the brain

8

Brain process language at different features
Hierarchical syntactic information occurs in the cortical zone situated between auditory-phonological and semantic zones.

Controlled experiments are task-based and not ecological

Matchin & Hickok et al. 2022

9 of 57

Ecologically valid?

9

10 of 57

10

11 of 57

Naturalistic stimuli

they are not constrained (controlled) in any way

allow for many different feature spaces to be applied to them

findings are more ecologically valid

Ecological validity: how well the findings from a controlled study would apply to the uncontrolled "real world".

you may need more fMRI data than in a conventional study when you are using a large feature space

11

12 of 57

Increasingly available open source ecological stimuli datasets

12

With advancement of ecological stimuli datasets and open source language models, recent studies looked at interesting open questions?

Deniz et al. 2019

Nastase et al. 2021

Lebel et al. 2022

Li et al. 2022

Zhang et al. 2021

?

How is information aggregated by the brain during language comprehension?

Is the “how” of the NLP system the same as “how” of the brain?

13 of 57

Brain Encoding and Decoding

13

How is the stimulus represented in the brain?

Reconstruct the stimulus, given the brain response?

Ivanova, Anna A., Martin Schrimpf, Stefano Anzellotti, Noga Zaslavsky, Evelina Fedorenko, and Leyla Isik. "Is it that simple? Linear mapping models in cognitive neuroscience." bioRxiv (2021).

14 of 57

Natural language is composed of many different features

De Heer et al. 2017

Feature representation

Low High

Huth et al. 2016; Deniz et al. 2019

Auditory Cortex

Broca’s Area

Visual Cortex

15 of 57

Mapping natural language to the human brain

15

16 of 57

What are we talking about when we talk about “mapping stimulus to the human brain”

I saw a magnificent picture in a book about the jungle, called True Stories.

Stimulus Space

Feature Space

Questions/Hypothesis

Mapping

Brain Activity Space

time

f

Encoding

	Concrete	Abstract
word₁	1	0
word₂	0	1
word₃	0	1

Manual labeling

17 of 57

What are we talking about when we talk about “mapping stimulus to the human brain”

I saw a magnificent picture in a book about the jungle, called True Stories.

Stimulus Space

Feature Space

Questions/Hypothesis

Mapping

Brain Activity Space

time

f

Encoding

Computational model

18 of 57

What are we talking about when we talk about “mapping stimulus to the human brain”

I saw a magnificent picture in a book about the jungle, called True Stories.

Stimulus Space

Feature Space

Questions/Hypothesis

Mapping

Brain Activity Space

time

f

Encoding

Concrete

Abstract

word₁

1

0

word₂

0

1

word₃₀

1

…

19 of 57

What are we talking about when we talk about “mapping stimulus to the human brain”

I saw a magnificent picture in a book about the jungle, called True Stories.

Stimulus Space

Feature Space

Questions/Hypothesis

Mapping

Brain Activity Space

time

f

Encoding

Decoding

Concrete

Abstract

word₁

1

0

word₂

0

1

word₃₀

1

…

20 of 57

I saw a magnificent picture in a book about the jungle, called True Stories.

Feature Space

Mapping

Brain Activity Space

time

f

Encoding

Concrete

Abstract

word₁

1

0

word₂

0

1

word₃₀

1

…

Models based on biological and physical foundations

Models based on prior knowledge and some experimental data�Models where no prior knowledge is available (data-driven)

Stimulus Space

White-box: Grey-box: Black-box:

Decoding

21 of 57

What we would like from a brain mapping approach

Provides a framework to test multiple hypotheses and models

Bridges between cognitive concepts and brain measurements

Creates fine grained cortical maps in individual subjects
Provides objective measurements of relevance (e.g. prediction accuracy, generalization to held-out data and subjects)

Provides interpretability of cortical maps and models

21

22 of 57

Voxelwise encoding modeling: Model training

Low-level

Extract stimulus feature for lexical semantics a and nuisance regressors (e.g. visual, word length)

Semantic

Syntactic

John hit the ball.

SUBJECT

ROOT

DET

OBJECT

23 of 57

Voxelwise encoding modeling: Model training

Semantic

ω

Low-level

Syntactic

John hit the ball.

SUBJECT

ROOT

DET

OBJECT

Regression weights map from feature space to brain responses.

! = argmin_![k(Y - X!)k²+ k(-\C!)k²]

2 2

24 of 57

Voxelwise encoding modeling: Model testing

Evaluate generalization to held-out dataset

ω

25 of 57

Voxelwise encoding modeling: Model testing

Record brain responses to test stimuli and

extract stimulus features.

ω

26 of 57

Voxelwise encoding modeling: Model testing

Predict brain responses to test stimuli.

ω

27 of 57

Voxelwise encoding modeling: Model testing

Compare timecourses of voxel predictions

and actual recordings separately for each language.

ω

28 of 57

Let’s code it! 👨‍💻👩‍💻

28

29 of 57

Low-level sensory features explain brain responses in modality-specific sensory regions

29

Deniz et al. 2019; Chen et al. 2024

30 of 57

Semantic representations are distributed across the cortex

Listening predicting Listening

Reading predicting Reading

LH

superior

p value (FDR) corrected

NS

0.05

0.001

10

-05

10

-09

10

-15

10

-21

0.0 0.5

PFC

MPC

LPC

LTC

VTC

PFC

MPC

LPC

LTC VTC

PFC

MPC

LPC

LTC

VTC

PFC

MPC

LPC

LTC VTC

Model Prediction Accuracy (r)

Deniz et al. 2019

31 of 57

Predict brain responses to held-out story across modalities

r

Recorded Brain Responses to Test Story

Predicted Brain Responses

Deniz et al. 2019

^ωreading

^ωlistening

32 of 57

Semantic representations are independent of the modality

Reading predicting Listening

Listening predicting Reading

PFC

MPC

LPC

LTC

VTC

PFC

MPC

LPC

PFC

MPC

LPC

LTC

VTC

PFC

MPC

LPC

LTC VTC

Model Prediction Accuracy (r)

p value (FDR) corrected

NS

0.05

0.001

10

-05

10

-09

10

-15

10

-21

0.0 0.5

LH

superior

Deniz et al. 2019

33 of 57

Replication of results across individual subjects and separate experiments

Listening predicting Reading Reading predicting Listening

Deniz et al. 2019

34 of 57

https://gallantlab.org/viewer-deniz-2019/

34

35 of 57

Do different brain regions represent linguistic information at different timescales?

35

36 of 57

Reliability of brain responses to stimulus scrambled at different timescales

36

Lerner et al. 2011

37 of 57

Reliability of brain responses to stimulus scrambled at different timescales

37

Lerner et al. 2011

Can we measure timescales under natural experimental conditions?

Are brain representations timescales similar across modalities?

38 of 57

Mapping semantic features to the human brain

38

39 of 57

Semantic Projection

39

40 of 57

What are word embeddings?

�

Words that appear in similar contexts (that is, they are semantically related) are located close together in the embedding space

��

Conceptual relations can be seen as geometrical relationships

Representations of words by vectors in high-dimensional space�

40

41 of 57

Limitations of (static) word embeddings

Representing the meaning of a word by 1 point�
Dimensions have no explicit meaning�
Measuring similarity as cosine similarity gives 1 single value

41

42 of 57

What is more similar?

42

43 of 57

Semantic concepts are multidimensional

If we compare them in size:

Dolphin and shark are more similar to each other than to whale�

If we compare them in dangerousness:

Dolphin and whale are more similar to each than to shark

Instead of comparing words as overall distance (e.g. cosine similarity), we should compare them along different dimensions�
But dimensions in the embedding space have no explicit meaning…�

43

44 of 57

44

45 of 57

How to compute the dimension of SIZE

Choose a list of typically large words and list of typically small words

[large, big, huge] vs [small, little, tiny]�

Take the difference between all large - small vector pairs and average them�
We can now project any word on this size dimension (e.g. dot product)

45

46 of 57

Evaluate projections onto different dimensions

Doesn’t work for all semantic categories

For each word, compare semantic projection with human rating�
The higher the correlation the better

46

47 of 57

Semantic projection on the brain

In voxelwise encoding models, the estimated model weights contain the semantic information reflected in each voxel�
We can use semantic projection with these weights to find out where a certain type of information is represented in the brain!�
In my research, I am interested in comparing abstract and concrete concepts (also across languages)...

47

48 of 57

Framework

Prediction accuracy

Model evaluation

Voxelwise encoding

Model estimation

Train stimulus

Test stimulus

Model weights

Train response

Test response

Prediction

Voxels

car

red

the

车

轿

红

法

想

新

Prediction accuracy

Voxels

Features

English

Chinese

Features

…

idea

new

the

…

car

red

the

…

车

轿

红

法

想

新

…

idea

new

the

…

Train response

49 of 57

Concreteness tuning

whether a brain regions represents predominantly abstract or concrete concept

49

English

Model

weights

life

fact

joy

phone

tree

house

Chinese

Projection

Voxels

Features

the model weights reflect the semantic information represented in each voxel for each language

compute abstract-to-concrete vector in semantic embedding space by averaging differences of word vector pairs

project model weights onto vector to get concreteness tuning for each voxel in each language

50 of 57

Concreteness tuning in English

posterior regions close to visual cortex are tuned towards concrete concepts�
language regions such as Broca’s area, high-level auditory cortex, superior temporal sulcus are tuned towards abstract concepts

50

English

superior

EVC

Broca

AC

MPC

PFC

AC

MPC

Broca

EVC

not well predicted in English: sqrt(R^2) < 0.1

51 of 57

Let’s code it! 👩‍💻👨‍💻

51

52 of 57

Brain Encoding and Decoding

52

How is the stimulus represented in the brain?

Reconstruct the stimulus, given the brain response?

Ivanova, Anna A., Martin Schrimpf, Stefano Anzellotti, Noga Zaslavsky, Evelina Fedorenko, and Leyla Isik. "Is it that simple? Linear mapping models in cognitive neuroscience." bioRxiv (2021).

53 of 57

What is Brain Decoding?

Can you read the mind with fMRI?
Or atleast tell what the person saw?

Visual Task

Language Task

Smith et al., 2011, Wang et al. 2019

54 of 57

Linguistic Decoding

input

output

Zou et al., 2022

55 of 57

Toward a universal decoder

Prior works decode linguistic meaning from imaging data has been largely limited to

concrete nouns,
using similar stimuli for training and testing,
small number of semantic categories.

Mitchell et al. 2008, Pereira et al. 2018

56 of 57

Goal: analyzing information represented in BOLD activity

Encoding

stimuli to predict activity.
how activity varies when there is concurrent variation in the world?

Decoding

activity to predict information about the stimuli.
how much can be learned about the world by observing activity?

56

57 of 57

Choose 2 papers!

1 to present

1 to review

Read all other abstracts!

57