How does the Brain Process Syntax Structure while Listening

Subba Reddy Oota¹, Veeral Agarwal², Mounika Marreddy²,

Manish Gupta^2,3, Bapi Raju Surampudi²

July 10, 2023

ACL 2023

¹Inria Bordeaux France, ²IIIT-Hyderabad, ³Microsoft India

Syntactic Parsing

• Syntax: how do words structurally combine to form sentences and meaning?
• In natural language processing, there are two popular syntactic parsing methods
• Constituency parsing
• Dependency parsing

What is fMRI?

A listening task in the scanner

Narrative Story

fMRI Brain Activity

https://www.biopac.com/events/fmri-psych/

Samuel A. Nastase. The “Narratives” fMRI dataset for evaluating models of naturalistic language comprehension. Nature, 2021.

The neurobiology of syntax and semantics has been a popular topic in language-fMRI studies

• Earlier language-fMRI encoding studies have observed that
• sentence semantics alone cannot explain all the variance in brain activity
• syntactic information can also be used to explain some of the variance
• Prior to different aspects of semantic interpretation, the brain performs syntactic structure analysis inherently
• e.g., we identify “Brazil”, “four”, “world cups”, and “2002” in a sentence: “Brazil won four world cups till 2002” before interpreting the semantics.
• More naturalistic fMRI datasets are increasingly available.
• Still unexplored whether different brain regions are associated with building different kinds of syntactic structures.

Two paradigms of syntactic parsing

• According to linguistic theories and recent linguistic studies, the two syntactic parsers capture distinct syntactic entities i.e. they are not strictly equivalent.
• Constituent syntactic structure:
• recursive grouping of sentence constituents (words and phrases)
• Dependencies:
• capture direct relations between words, identical to thematic functions such as subject, object, modifier, etc.
• Hence, dependency and constituent structures are distinct and the type of information they capture is nonequivalent.
• Questions:
• Do syntactic parser representations predict similar fMRI areas?
• Which syntactic parsing method better predicts fMRI activity?

Related Studies

6

Wang et al. 2020, Caucheteux et al. 2021, Zhang et al. 2022, Reddy et al. 2022

Contributions

• We explore syntactic structure embeddings obtained from three parsers and use them in an encoding model to predict brain responses.
• We use a GCN model (SynGCN embeddings) for the dependency parser that accurately encodes the global syntactic information.
• We show that constituency and dependency tree-based embeddings are effective across different language regions for brain activity prediction.
• We show that both syntactic embeddings explain additional variance even after controlling for basic syntactic signals.
• We find that
• Bilateral temporal areas (ATL, PTL) and middle-frontal gyrus (MFG) are significantly related to constituency parse representations.
• The angular gyrus (AG) and posterior cingulate cortex (PCC) are significantly associated with dependency parse embeddings.

​

Brain Encoding?

Present

Stimulus

Stimulus

Ridge Regression

Input

Input

Output

X

Y

W

Explained Variance (R²)

Encoding: training independent models

• Independent model per participant

• Independent model per voxel / sensor-timepoint

Workflow

Listening data target: human brain recordings

11

• We use Pieman story listening:
• 82 subjects,
• 282 TRs (repetition time)
• here it is 1.5 sec.

​

Example: ''I began my illustrious carrier in journalism…''

Samuel A. Nastase. The “Narratives” fMRI dataset for evaluating models of naturalistic language comprehension. Nature, 2021.

Feature Representaions

Semantic Embeddings

Syntactic Tree Parsers

​

Complexity Metrics

Basic Syntax Features

Punctuations (PU)

​

Part-of-Speech & Depenency Tags (PD)

Word Frequency

Node Count

Word Length

Constituent Complete (CC)

​

Constituent Incomplete (CI)

​

Incremental Top-Down (INC)

​

Dependecy (DEP)

BERT

Constituent Complete Trees: The largest subtrees completed by a few of the words in the sentence

Constituent InComplete Trees: The largest subtrees completed by a few of the words in the sentenc

• Retaining all the phrase structure grammar productions that are required to derive the words seen till then, starting from the root of the sentence’s tree.

Incremental Top-Down Trees: processes input strings from left to right

Building Syntactic Tree Embeddings (CC, CI and INC)

Reddy, Aniketh Janardhan, and Leila Wehbe. "Can fMRI reveal the representation of syntactic structure in the brain?." Advances in Neural Information Processing Systems 34 (2021): 9843-9856.

Building Syntactic Tree Embeddings (DEP)

Semantic Word Embeddings: BERT

Transformer language models

(BERT, XLM, GPT,…)

Vaswani et al. 2017, Devlin et al. 2019

​

Experimental Setup and Evaluation Metric

• 4-fold (K=4) cross-validation
• R²-score
• Statistical Significance:
• We run a permutation test to check if R²-scores are significantly higher than chance
• The predictions are permuted within fold 5000 times
• The FDR correction is performed by grouping all the voxel-level p-values (i.e., across all subjects and feature groups) and choosing one threshold for all of our results.

​

Performance of individual embedding methods

Brain Prediction Results

Additional Predictive Power of various Representations

Pairwise predictive power comparison for syntactic parse methods and BERT

Findings & Future Works

​

​

​

​

• This work was done on data related to English stories only.
• For languages in other language families, syntactic structure may be very different from English.
• Hence, more work needs to be done to check which of these insights hold for datasets in other language families.
• More work needs to be done to design representations (like prosodic features) for auditory stimuli.

Constituency tree structure is better in temporal cortex and MFG, while Dependency structure is better in AG and PCC.

​

Regions predicted by syntactic and semantic are difficult to distinguish

Collaborators

Subba Reddy Oota

Mounika Marreddy

Manish Gupta

Bapi Raju Surampudi