Subba Reddy Oota Akshett Jindal Ishani Mondal Khushbu Pahwa

Satya Sai Srinath Manish Shrivastava Maneesh Singh Bapi S. Raju Manish Gupta

​

Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain)

subba.reddy.oota@tu-berlin.de

2

Language models (LMs) are trained to predict missing words

Language model

The

quick

brown

fox

[MASK]

jumps

3

Language models (LMs) predict brain activity evoked by complex

language (e.g. listening a story) to an impressive degree

time

Brain alignment of an LM ⇒ how similar its representations are to a human brain

4

Language models (LMs) predict brain activity evoked by complex

language (e.g. listening a story) to an impressive degree

Jain and Huth. Incorporating context into language encoding models for fMRI. (NeurIPS 2018)

Toneva and Wehbe. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). (NeurIPS 2019)

Brain alignment of a LM ⇒ Advances in model size, instruction-tuning, and multimodality have improved alignment with neural data.

brain alignment_i = Pearson corr(true v_i, pred v_i)

Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, Enhong Chen. "A Survey on Multimodal Large Language Models." Arxiv 2024.

5

Multimodal instruction tuning enables models to generalize to new tasks by following unseen instructions

Do multimodal instruction-tuned models prompted with natural language improve brain alignment and capture instruction-specific representations?

How does the brain integrate information during the processing of visual images?

?

How do multimodal instruction-tuned LLMs process visual images when guided by natural language task instructions?

6

Multi-modal Instruction-tuned LLMs (MLLMs): brain alignment

Task-specific instructions

Image Captioning:

What is the caption of the image?

Image Understanding:

Describe the most dominant color in the image.

Visual Relationship:

What objects are being used by the largest animal in this image?

Multimodal instruction-tuned model (MLLM)

Instruction Embedding

estimate alignment

Early Visual

• How well do MLLMs predict brain activity evoked by visual stimuli under task-specific instructions compared to unimodal and multimodal models?
• Do instruction-specific representations in MLLMs differentiate visual brain regions involved in processing, thereby aligning with the mechanisms of human visual cognition?

7

Datasets & Models

• Brain: fMRI recordings from NSD dataset [St-Laurent et al. 2023]
• Passively watching natural scene images
• N=4

​

• 3 multimodal instruction-tuned large language models
• InstructBLIP
• mPLUG-Owl
• IDEFICS

​

• unimodal and multi-modal models
• ViT-H
• CLIP

To quantify model predictions, we have an estimate of the explainable variance and use that to measure normalize brain alignment.

8

Task-specific natural instructions

These tasks which are generally applicable to any image regardless of the contents in the image

9

How do MLLMs, unimodal and multi-modal models differ in their ability to predict brain activity in higher visual and early visual regions?

Result-1: MLLMs vs. Unimodal vs. Multi-modal models and brain alignment

10

• Early-visual regions
• Both MLLMs and multi-modal models show significantly high brain alignment than baseline and unimodal video models
• Surprisingly, brain alignment of random initialization of MLLMs is closer to that of unimodal video models
• Higher-visual regions
• Both MLLMs and multi-modal models show better brain relevant representations (∼0.8) than early visual areas (∼0.6).

11

• Early-visual regions
• Both MLLMs and multi-modal models show significantly high brain alignment than baseline and unimodal video models
• Surprisingly, brain alignment of random initialization of MLLMs is closer to that of unimodal video models
• Higher-visual regions
• Both MLLMs and multi-modal models show better brain relevant representations (∼0.8) than early visual areas (∼0.6).

Which task-specific instructions are highly correlated to visual function localizers?

Result-1: MLLMs vs. Unimodal vs. Multi-modal models and brain alignment

12

Result-2: Which task-specific instructions are highly correlated to visual function localizers?

S1: InstructBLIP

S1: mPLUG-Owl

• Early-visual regions
• Image understanding instruction shows significantly high brain alignment across MLLMs
• Higher-visual regions
• Image captioning instruction shows significantly high brain alignment in the EBA, PPA, and FFA regions
• Visual question answering instructions shows significantly high brain alignment in the PPA, and FFA regions
• Not all instructions lead to increased brain alignment across all regions

13

Result-2: Which task-specific instructions are highly correlated to visual function localizers?

S1: InstructBLIP

S1: mPLUG-Owl

• Early-visual regions
• Image understanding instruction shows significantly high brain alignment across MLLMs
• Higher-visual regions
• Image captioning instruction shows significantly high brain alignment in the EBA, PPA, and FFA regions
• Visual question answering instructions shows significantly high brain alignment in the PPA, and FFA regions
• Not all instructions lead to increased brain alignment across all regions

Do task-specific instructions from MLLMs account for visual concepts understanding?

14

Result-3: MLLMs capture count- and recognition-related visual concepts effectively across instructions

• Visual concept-Count
• VQ2 instruction shows significantly high brain alignment in high-level visual regions, while IU2 and IU3 instructions show higher alignment in early visual regions
• Visual concept-Recognition
• Both VQ1 and VQ2 instruction show significantly high brain alignment across high-level and early-visual regions

15

What is the unique and shared variance of each task-specific instruction to brain responses?

16

Result-4: Partitioning explained variance between task-specific instructions

• Between Image Captioning (IC) and Image Understanding (IU2): there is no unique variance for IU2 in the EBA region (higher-visual), while IC retains some unique variance.
• Task-specific instructions exhibit moderate shared variance in the early visual cortex, while shared variance is significantly higher in higher visual ROIs

17

Conclusions

1. 🗣️ MLLMs generate task-specific output tokens based on instructions, but not all instructions lead to better brain alignment

​

• 👁️‍🗨️ They capture multiple visual concepts, yet exhibit similar brain alignment across different types of visual stimuli

​

• The variance in brain alignment is shared across task-specific instructions:�▸ Moderate in 🧠 early visual areas�▸ Higher in 🧠 high-level visual regions

​

• But more work to do - especially in enhancing MLLMs’ ability to differentiate between instruction types in terms of neural alignment

18

Subba Reddy Oota

Correlating instruction-tuning (in multimodal models) with vision-language

processing (in the brain) (ICLR-2025)

Manish Gupta

Bapi S. Raju

Satya Sai Srinath

Khushbu Pahwa

Maneesh Singh

Ishani Mondal

Manish Shrivastava

Akshett Jindal