Subba Reddy Oota Vijay Rowtula Satya Sai Srinath Khushbu Pahwa

Anant Khandelwal Manish Gupta Tanmoy Chakraborty Bapi S. Raju

​

Linguistic Properties and Model Scale in Brain Encoding: From Small to Compressed Language Models

2

Language models (LMs) predict brain activity evoked by complex

language (e.g. listening a story) to an impressive degree

time

Brain alignment of a LM ⇒ how similar its representations are to a human brains

3

Language models (LMs) predict brain activity evoked by complex

language (e.g. listening a story) to an impressive degree

Jain and Huth. Incorporating context into language encoding models for fMRI. (NeurIPS 2018)

Toneva and Wehbe. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). (NeurIPS 2019)

Brain alignment of a LM ⇒ does better brain alignment demand scaling of language models?

brain alignment_i = Pearson corr(true v_i, pred v_i)

4

Antonello et al. 2023 NeurIPS

LMs predict brain activity, and that alignment improves as models scale, suggesting neural scaling laws

Larger models, better brain alignment, but harder to interpret and costly. How small can we go?

5

How small can we go? Three questions about scale, compression, and the brain

Use model size and precision as controlled "knobs" to probe what representations the brain needs.

1. What's the minimal model capacity that matches large LLMs in brain alignment, for both encoding and decoding?

​

• How do compression methods (quantization, pruning) affect brain alignment in small and large models?

​

• Which linguistic properties survive or degrade across small/large/compressed models, and do those changes impact brain alignment?

​

6

Two knobs: model scale and numerical precision

Investigate via probing

Naturalistic stimulus

Brain alignment shifts with scale and compression - does linguistic competence shift the same way?

7

Does linguistic competence drive brain–model alignment?

Investigate via an indirect approach

Morphology, Discourse, Semantics, Syntax, Reasoning

Naturalistic stimulus

Linguistic Competence

8

Datasets & Model

• Brain: fMRI recordings from Subset-Moth-Radio-Hour [Deniz et al. 2019]
• Reading & Listening to the same short stories
• N=9

​

• 3 language model families
• Qwen-2.5 (1.5B, 3B, 7B, 14B)
• LLaMA-3.2 (1B, 3B, 8B, 14B)
• DeepSeek-R1 (1.5B, 3B, 7B, 14B)

​

• 2 compression methods
• Quantization (AWQ, GPTQ, SmoothQuant)
• Unstructured Pruning (10%, 25%, 50%)

​

To quantify model predictions, we have an estimate of the explainable variance and use that to measure normalize brain alignment.

9

What is the minimal model capacity required to achieve brain alignment comparable to larger LLMs?

Result-1: SLMs vs. LLMs vs. Quantization variants & brain alignment

• 3B SLMs match brain alignment with 7B-14B LLMs, but 1.5B SLMs consistently underperform
• Compression analyses show that brain alignment is largely robust to post-training efficiency methods.

Is brain alignment equally robust to all compression methods?

11

How do compression methods (quantization and pruning) affect brain alignment for both SLMs and LLMs?

Result-2: Brain alignment is robust to compression, except GPTQ

Moderate pruning (10–25%) preserves alignment at both scales (1.5-3B), but aggressive 50% pruning collapses the 1.5B model

Does linguistic competence vary across scale and precision?

13

Which linguistic properties are preserved or degraded across small, large, and compressed models, and do these changes correlate with brain alignment?

Result-3: Linguistic competence and brain alignment dissociate

• 3B and 7B models cluster at the top of brain alignment, competence and alignment come apart
• AWQ and SmoothQuant preserve both competence and alignment; GPTQ reduces both
• Tiny SLMs (1.5B) can keep the linguistic properties yet lose brain-relevant representations

Conclusions for neuro-AI research field

1. Brain alignment saturates early (~3B): 3B SLMs match 7B–14B LLMs, while 1B–1.5B models reduce alignment, especially in semantic regions.

​

• Alignment is robust to compression—except GPTQ: AWQ and SmoothQuant preserve alignment; GPTQ consistently degrades it.

​

• Linguistic competence and brain alignment dissociate: Compression can degrade specific linguistic properties without impacting alignment, while 1B–1.5B models keep linguistic properties yet fail to capture brain-relevant representations.

​

16

Subba Reddy Oota

Linguistic Properties and Model Scale in Brain Encoding: From Small to Compressed Language Models (ICML-2026)

Manish Gupta

Bapi S. Raju

Anant Khandelwal

Khushbu Pahwa

Satya Sai Srinath

Vijay Rowtula

Tanmoy Chakraborty