Brain-Inspired AI 2.0: Aligning Language Models Across Languages and Modalities

Subba Reddy Oota¹, Tanmoy Chakraborty², Manish Gupta^3,4, Raju S. Bapi³

​

¹TU Berlin, Germany; ²IIT Delhi, India; ³IIIT Hyderabad, India; ⁴Microsoft, India

​

subba.reddy.oota@tu-berlin.de, tanchak@iitd.ac.in, gmanish@microsoft.com, raju.bapi@iiit.ac.in

AAAI 2026: Brain-Inspired AI 2.0

Agenda

• Introduction to the tutorial [10 min]
• Introduction to Brain Encoding and Decoding [50 min]
• Brain Encoding: Scaling Laws, Multilinguality, Multimodal and Instruction-tuned Models [60 min]
• Coffee Break & Networking [30 min]
• Brain-informed Fine-tuning of Language Models [30 min]
• Brain-based Interpretability and Causal Testing of AI Models [20 min]
• Brain Decoding [30 min]
• Summary and Future Trends [10 min]

​

AAAI 2026: Brain-Inspired AI 2.0

Agenda

• Introduction to the tutorial [10 min]
• Introduction to Brain Encoding and Decoding [50 min]
• Brain Encoding: Scaling Laws, Multilinguality, Multimodal and Instruction-tuned Models [60 min]
• Coffee Break & Networking [30 min]
• Brain-informed Fine-tuning of Language Models [30 min]
• Brain-based Interpretability and Causal Testing of AI Models [20 min]
• Brain Decoding [30 min]
• Summary and Future Trends [10 min]

​

AAAI 2026: Brain-Inspired AI 2.0

Agenda

• Introduction to the tutorial [10 min]
• Introduction to Brain Encoding and Decoding [50 min]
• Brain Encoding/Decoding and applications
• Introduction to popular datasets
• Text Stimulus Representations
• Alignment Between AI Models and Human Brain Language Comprehension
• Brain Encoding: Scaling Laws, Multilinguality, Multimodal and Instruction-tuned Models [60 min]
• Coffee Break & Networking [30 min]
• Brain-informed Fine-tuning of Language Models [30 min]
• Brain-based Interpretability and Causal Testing of AI Models [20 min]
• Brain Decoding [30 min]
• Summary and Future Trends [10 min]

​

AAAI 2026: Brain-Inspired AI 2.0

Agenda

• Introduction to the tutorial [10 min]
• Introduction to Brain Encoding and Decoding [50 min]
• Brain Encoding/Decoding and applications
• Introduction to popular datasets
• Text Stimulus Representations
• Alignment Between AI Models and Human Brain Language Comprehension
• Brain Encoding: Scaling Laws, Multilinguality, Multimodal and Instruction-tuned Models [60 min]
• Coffee Break & Networking [30 min]
• Brain-informed Fine-tuning of Language Models [30 min]
• Brain-based Interpretability and Causal Testing of AI Models [20 min]
• Brain Decoding [30 min]
• Summary and Future Trends [10 min]

​

AAAI 2026: Brain-Inspired AI 2.0

Neuroscience

• Field of science that studies the structure and function of the nervous system of different species.
• How is information represented and processed in the brain?
• How does the brain integrate text, vision, sound, and touch?
• How do abstract concepts (like numbers or language) get encoded?
• How does the brain learn and change?
• What are the neural mechanisms that allow learning, memory, and adaptation?
• How are memories formed but not overwritten?
• How does the brain learn from very few examples?
• How does RL work biologically?
• How do large-scale brain circuits coordinate behavior?
• How do distributed brain regions work together to produce coherent behavior?
• How does the brain decide when to act or inhibit action?
• How do sensory inputs get transformed into motor outputs?

AAAI 2026: Brain-Inspired AI 2.0

Neuroscience

• What goes wrong in neurological and psychiatric disorders, and how can we fix it?
• How do disruptions in brain structure or function cause diseases (Alzheimer’s, depression, Parkinson’s, autism, and schizophrenia), and how can we treat them?
• Precision psychiatry: Can we predict which treatment will work for whom, when, and why?
• How does the brain give rise to the mind and consciousness?
• How do electrical and chemical signals in neurons produce subjective experiences like thoughts, emotions, awareness, and the sense of self?
• Is consciousness localized or distributed?
• Can machines ever be conscious?
• What changes in the brain when consciousness is lost?
• Brain-computer interfaces

AAAI 2026: Brain-Inspired AI 2.0

Brain encoding and decoding in cognitive neuroscience

Ivanova, Anna A., Martin Schrimpf, Stefano Anzellotti, Noga Zaslavsky, Evelina Fedorenko, and Leyla Isik. "Is it that simple? Linear mapping models in cognitive neuroscience." bioRxiv (2021).

8

20-Jan-26

AAAI 2026: Brain-Inspired AI 2.0

Brain encoding and decoding

AAAI 2026: Brain-Inspired AI 2.0

Techniques for studying the brain function

• fMRI: high spatial but low time resolution.
• Good to study a specific location in the brain
• Unsuitable for sentence-level analysis. fMRI takes about two seconds to complete a scan. This is far lower than the speed at which humans can process language.
• Cannot capture syntactic information (Gauthier and Levy, 2019)
• EEG: high time but low spatial resolution.
• Can preserve rich syntactic information (Hale et al., 2018)
• But cannot use for source analysis.
• fNIRS: compromise option
• Time resolution better than fMRI
• Spatial resolution better than EEG
• Balance of spatial and temporal resolution may not be enough to compensate for the loss in both.

Vogel, Jörn, Sami Haddadin, Beata Jarosiewicz, John D. Simeral, Daniel Bacher, Leigh R. Hochberg, John P. Donoghue, and Patrick van der Smagt. "An assistive decision-and-control architecture for force-sensitive hand–arm systems driven by human–machine interfaces." The International Journal of Robotics Research 34, no. 6 (2015): 763-780.

10

20-Jan-26

Single Micro-Electrode (ME), Micro-Electrode array (MEA), Electro-Cortico Graphy (ECoG), Positron emission tomography (PET), functional MRI (fMRI), Magneto-encephalography (MEG), Electro-encephalography (EEG), Near-Infrared Spectroscopy (NIRS)

AAAI 2026: Brain-Inspired AI 2.0

Computational Cognitive Science Research goals

• Predictive Accuracy
• Compare feature sets: Which feature set provides the most faithful reflection of the neural representational space?
• Test feature decodability: “Does neural data Y contain information about features X?”
• Build accurate models of brain data: Aim is to enable simulations of neuroscience experiments.
• Interpretability
• Examine individual features: Which features contribute the most to neural activity?
• Test correspondences between representational spaces
• “CNNs vs ventral visual stream” or “Two text representations”
• Interpret feature sets
• Do features X, generated by a known process, accurately describe the space of neural responses Y?
• Do voxels respond to a single feature or exhibit mixed selectivity?
• How does the mapping relate to other models or theories of brain function?
• …

Ivanova, Anna A., Martin Schrimpf, Stefano Anzellotti, Noga Zaslavsky, Evelina Fedorenko, and Leyla Isik. "Is it that simple? Linear mapping models in cognitive neuroscience." bioRxiv (2021).

AAAI 2026: Brain-Inspired AI 2.0

Agenda

• Introduction to the tutorial [10 min]
• Introduction to Brain Encoding and Decoding [50 min]
• Brain Encoding/Decoding and applications
• Introduction to popular datasets
• Text Stimulus Representations
• Alignment Between AI Models and Human Brain Language Comprehension
• Brain Encoding: Scaling Laws, Multilinguality, Multimodal and Instruction-tuned Models [60 min]
• Coffee Break & Networking [30 min]
• Brain-informed Fine-tuning of Language Models [30 min]
• Brain-based Interpretability and Causal Testing of AI Models [20 min]
• Brain Decoding [30 min]
• Summary and Future Trends [10 min]

​

AAAI 2026: Brain-Inspired AI 2.0

Types of stimuli and popular datasets

• Text (Words, Sentences, Paragraphs): Harry Potter Story, ZUCO EEG, Question-Answering MEG.
• Visual: Binary visual patterns, Natural Images (Vim-1), BOLD5000, Algonauts and SS-fMRI.
• Audio: Alice’s Adventures in Wonderland, Narratives, The Moth Radio Hour, Audio stories.
• Videos: BBC’s Doctor Who, Japanese Ads, Pippi Langkous, Algonauts.
• Other Multimodal Stimuli: Words + line drawing of concept named by each word, Pereira.

AAAI 2026: Brain-Inspired AI 2.0

Forms of stimulus presentation and data collection

• Type: fMRI, EEG, MEG, …
• TR: Sampling time.
• Fixation points: location, color, shape.
• Form of stimuli presentation: text, video, audio, images.
• Task: question answering, property generation, understanding, …
• Time given to participants: 1 minute to list properties, …
• Type of participants: males/females, sighted/blind, …
• Number of times the response to stimuli was recorded.
• Stimulus Language

AAAI 2026: Brain-Inspired AI 2.0

Text Stimulus Datasets

AAAI 2026: Brain-Inspired AI 2.0

Visual Stimulus Datasets

AAAI 2026: Brain-Inspired AI 2.0

Audio Stimulus Datasets

AAAI 2026: Brain-Inspired AI 2.0

Video Stimulus Datasets

AAAI 2026: Brain-Inspired AI 2.0

Other Multimodal Stimulus Datasets

AAAI 2026: Brain-Inspired AI 2.0

Agenda

• Introduction to the tutorial [10 min]
• Introduction to Brain Encoding and Decoding [50 min]
• Brain Encoding/Decoding and applications
• Introduction to popular datasets
• Text Stimulus Representations
• Alignment Between AI Models and Human Brain Language Comprehension
• Brain Encoding: Scaling Laws, Multilinguality, Multimodal and Instruction-tuned Models [60 min]
• Coffee Break & Networking [30 min]
• Brain-informed Fine-tuning of Language Models [30 min]
• Brain-based Interpretability and Causal Testing of AI Models [20 min]
• Brain Decoding [30 min]
• Summary and Future Trends [10 min]

​

AAAI 2026: Brain-Inspired AI 2.0

Text Stimulus Representations

• Basic NLP Representations
• Corpus co-occurrence counts
• Topic models
• Linguistic: POS, dependencies, roles.
• Discourse
• Characters, motion, speech, emotions, non-motion verbs
• Deep Learning based Representations
• Embeddings
• Longer context using LSTMs
• Transformers
• Experiential attributes
• Rated on 0-6 scale
• Binary

AAAI 2026: Brain-Inspired AI 2.0

Basic NLP Representations for Word Stimuli

• Corpus co-occurrence counts
• 25 verbs (Mitchell et al., 2008; Pereira et al., 2013)
• Basic sensory and motor activities, actions performed on objects, and actions involving changes to spatial relationships.
• Verbs: see, hear, listen, taste, smell, eat, touch, nib, lift, manipulate, run, push, fill, move, ride, say, fear, open, approach, near, enter, drive, wear, break, and clean.
• For each (verb, stimulus word w), feature value = normalized co-occurrence count of w with any of three forms of the verb (e.g., taste, tastes, or tasted) over the text corpus.
• 985 common English words (such as above, worry, and mother) in (Huth et al., 2016).

• Topic models (Pereira et al., 2013)
• Get relevant Wiki pages (e.g., “airplane” is “Fixed-Wing Aircraft”) and other linked pages (e.g. “Aircraft cabin”)
• LDA topic modelling on 3500 pages with #topics from 10 to 100, in increments of 5, setting the α parameter to 25/#topics.
• LSA topic modelling (Wang et al., 2017)

​

​

22

20-Jan-26

AAAI 2026: Brain-Inspired AI 2.0

Basic NLP Representations for Word Stimuli

• Word length
• Is the word related to one of the 28 unique parts of speech and 17 unique dependency relationships?
• Position of word in the sentence
• Roles
• Main verb
• Agent or experiencer
• Patient or recipient
• Predicate of a sentence (The window was dusty)
• Modifier (The angry activist broke the chair)
• Complement in adjunct and propositional phrase, including direction, location, and time (The restaurant was loud at night).

Wehbe, Leila, Brian Murphy, Partha Talukdar, Alona Fyshe, Aaditya Ramdas, and Tom Mitchell. "Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses." PloS one 9, no. 11 (2014): e112575.

Wang, Jing, Vladimir L. Cherkassky, and Marcel Adam Just. "Predicting the brain activation pattern associated with the propositional content of a sentence: modeling neural representations of events and states." Human brain mapping 38, no. 10 (2017): 4865-4881.

​

​

AAAI 2026: Brain-Inspired AI 2.0

Discourse features (for Harry Potter dataset)

• Characters: Binary features to signal which of the 10 characters are mentioned.
• Motions: Identify a set of motions that occurred frequently in the chapter (e.g. fly, manipulate, collide physically, etc.).
• Speech: Indicate the parts of the story that correspond to direct speech between the characters. Used the presence of dialog as a feature. 
• Emotions: Identified a set of emotions that were felt by the characters in the chapter (e.g. annoyance, nervousness, pride, etc.).
• Verbs: Identified a set of actions that occurred frequently in the chapter that were distinct from motion (e.g. hear, know, see, etc.).

Wehbe, Leila, Brian Murphy, Partha Talukdar, Alona Fyshe, Aaditya Ramdas, and Tom Mitchell. "Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses." PloS one 9, no. 11 (2014): e112575.

Wang, Jing, Vladimir L. Cherkassky, and Marcel Adam Just. "Predicting the brain activation pattern associated with the propositional content of a sentence: modeling neural representations of events and states." Human brain mapping 38, no. 10 (2017): 4865-4881.

​

​

AAAI 2026: Brain-Inspired AI 2.0

DL Representations: Using embeddings for word stimuli

• GloVe 300D vectors (Pereira et al., 2016; Wang et al., 2017; Pereira et al., 2018; Anderson et al., 2019)
• 300D embeddings by training a skip-gram model using negative sampling (SGNS) on Italian and English Wikipedia dumps using Gensim. (Anderson et al., 2017a)
• FastText (Berezutskaya et al., 2020)
• Comparison across multiple embedding methods
• GloVe, word2vec, WordNet2Vec, FastText, ELMo (Hollenstein et al., 2019)
• word2Vec, fastText, GloVe, Dependency-based word2vec, RWSGwn, ConceptNet, ELMo, averaged and concatenated combinations (Wang et al., 2020)

AAAI 2026: Brain-Inspired AI 2.0

DL Representations: Using longer context for word stimuli

• Multi-task LSTMs
• Predict next word and POS of next word.

Toneva, Mariya, and Leila Wehbe. "Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)." Advances in Neural Information Processing Systems 32 (2019).

Jain, Shailee, and Alexander Huth. "Incorporating context into language encoding models for fMRI." Advances in neural information processing systems 31 (2018).

Jat, Sharmistha, Hao Tang, Partha Talukdar, and Tom Mitchell. "Relating simple sentence representations in deep neural networks and the brain." arXiv preprint arXiv:1906.11861 (2019).

​

​

• ELMo embeddings: LSTM based pretrained language model

​

26

20-Jan-26

AAAI 2026: Brain-Inspired AI 2.0

DL Representations: Using sentence embeddings

• Unstructured Models: Ignore sentence structure
• Simple Pooling Methods
• Average/max/concat(max, avg) pooling over word embeddings.
• Advanced Pooling Methods
• FastSent (Hill, Cho, and Korhonen 2016)
• SIF (Arora, Liang, and Ma 2016) adapts the naïve averaging of word embeddings to weighted averaging.
• Structured Models
• Unsupervised Methods: Skip-thought, QuickThought.
• Supervised Methods: InferSent, GenSen (Subramanian et al. 2018), Universal Sentence Encoder

​

Toneva, Mariya, and Leila Wehbe. "Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)." Advances in Neural Information Processing Systems 32 (2019).

Sun, Jingyuan, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. "Towards sentence-level brain decoding with distributed representations." In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, pp. 7047-7054. 2019.

​

​

​

27

20-Jan-26

AAAI 2026: Brain-Inspired AI 2.0

DL Representations: Transformer-based methods for text stimuli (Layer #, context length, architecture)

Toneva, Mariya, and Leila Wehbe. "Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)." Advances in Neural Information Processing Systems 32 (2019).

Sun, Jingyuan, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. "Neural encoding and decoding with distributed sentence representations." IEEE Transactions on Neural Networks and Learning Systems 32, no. 2 (2020): 589-603.

​

​

​

28

20-Jan-26

Transformer-XL is the only model that continues to increase performance as the context length is increased. In all networks, the middle layers perform the best for contexts longer than 15 words. The deepest layers across all networks show a sharp increase in performance at short-range context (fewer than 10 words), followed by a decrease in performance. [Toneva and Wehbe, 2019]

AAAI 2026: Brain-Inspired AI 2.0

DL Representations: Transformer-based methods for text stimuli (NLP task finetuning)

Oota, Subba Reddy, Jashn Arora, Veeral Agarwal, Mounika Marreddy, Manish Gupta, and Bapi Raju Surampudi. "Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity?." arXiv preprint arXiv:2205.01404 (2022).

​

29

20-Jan-26

Tasks

Paraphrase, Summarization, Question Answering, Sentiment Analysis, NER, Word Sense Disambiguation, Natural Language Inference, Semantic Role Labeling, Coreference Resolution, Shallow Syntax Parsing

Pereira dataset: CR, NER, and SS perform the best.

Dendrogram constructed using similarity on representations from task-specific Transformer encoder models with stimuli from the dataset passed as input.

AAAI 2026: Brain-Inspired AI 2.0

DL Representations: Transformer-based methods for text stimuli (Multi-task setup)

Schwartz, Dan, Mariya Toneva, and Leila Wehbe. "Inducing brain-relevant bias in natural language processing models." Advances in neural information processing systems 32 (2019).

​

• Settings
• Finetune BERT vs not
• Finetune BERT using one representative subject and train dense layer for each subject, vs finetune BERT for each subject.
• Finetune BERT on MEG for all subjects, then finetune BERT on fMRI.
• Multi-task finetune BERT for fMRI+MEG prediction task
• Results
• Fine-tuned models predict fMRI data better than vanilla BERT
• Using MEG data can improve fMRI predictions.
• A single model can be used to predict fMRI activity across multiple experiment participants.

30

20-Jan-26

AAAI 2026: Brain-Inspired AI 2.0

Experiential attributes model for text stimuli

• Represents words in terms of human ratings of their degree of association with different attributes of experience
• “On a scale of 0 to 6, to what degree do you think of a banana as having a characteristic or defining color?”
• Anderson et al., 2019: 65 attributes spanning sensory, motor, affective, spatial, temporal, causal, social, and abstract cognitive experiences.
• Value-add on top of text models: a lot of experiential information goes unstated in natural verbal communication.
• E.g., it is rarely useful to communicate the color of bananas because it is obvious to all those with experience of bananas.
• E.g., it would be unusual to specify that dropping things involves movement.
• Nishida et al., 2020 use a subset of 20 attributes.

Anderson, Andrew James, Jeffrey R. Binder, Leonardo Fernandino, Colin J. Humphries, Lisa L. Conant, Rajeev DS Raizada, Feng Lin, and Edmund C. Lalor. "An integrated neural decoder of linguistic and experiential meaning." Journal of Neuroscience 39, no. 45 (2019): 8969-8987.

Anderson, Andrew James, Jeffrey R. Binder, Leonardo Fernandino, Colin J. Humphries, Lisa L. Conant, Mario Aguilar, Xixi Wang, Donias Doko, and Rajeev DS Raizada. "Predicting neural activity patterns associated with sentences using a neurobiologically motivated model of semantic representation." Cerebral Cortex 27, no. 9 (2017): 4379-4395.

Anderson, Andrew James, Kelsey McDermott, Brian Rooks, Kathi L. Heffner, David Dodell-Feder, and Feng V. Lin. "Decoding individual identity from brain activity elicited in imagining common experiences." Nature communications 11, no. 1 (2020): 1-14.

​

​

31

20-Jan-26

AAAI 2026: Brain-Inspired AI 2.0

Binary attribute representations

• Each stimulus is represented using a binary vector capturing membership to one of the eight semantic categories.

​

​

Handjaras, Giacomo, Emiliano Ricciardi, Andrea Leo, Alessandro Lenci, Luca Cecchetti, Mirco Cosottini, Giovanna Marotta, and Pietro Pietrini. "How concepts are encoded in the human brain: a modality independent, category-based cortical organization of semantic knowledge." Neuroimage 135 (2016): 232-242.

Wang, Jing, Vladimir L. Cherkassky, and Marcel Adam Just. "Predicting the brain activation pattern associated with the propositional content of a sentence: modeling neural representations of events and states." Human brain mapping 38, no. 10 (2017): 4865-4881.

​

​

​

• 42 neurally plausible semantic features (NPSFs)
• 10 Perceptual and affective characteristics of an entity. E.g, man-made, size, color, temperature, positive affective valence, high affective arousal
• Animate beings (person, human-group, animal)
• Time and space properties (e.g. unenclosed setting, change of location)

​

32

20-Jan-26

AAAI 2026: Brain-Inspired AI 2.0

Agenda

• Introduction to the tutorial [10 min]
• Introduction to Brain Encoding and Decoding [50 min]
• Brain Encoding/Decoding and applications
• Introduction to popular datasets
• Text Stimulus Representations
• Alignment Between AI Models and Human Brain Language Comprehension
• Brain Encoding: Scaling Laws, Multilinguality, Multimodal and Instruction-tuned Models [60 min]
• Coffee Break & Networking [30 min]
• Brain-informed Fine-tuning of Language Models [30 min]
• Brain-based Interpretability and Causal Testing of AI Models [20 min]
• Brain Decoding [30 min]
• Summary and Future Trends [10 min]

​

AAAI 2026: Brain-Inspired AI 2.0

Does context size impact alignment?

• ELMo, BERT, USE (Universal Sentence Encoder) and T-XL.
• Transformer-XL continues to increase perf as the context length is increased.
• Middle layers perform the best for contexts of 15+ words.
• Deepest layers show a sharp increase in perf at short-range context (<10 words).

​

Toneva, Mariya, and Leila Wehbe. "Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)." Advances in neural information processing systems 32 (2019).

34

20-Jan-26

AAAI 2026: Brain-Inspired AI 2.0

What if we use uniform attention in first few layers?

• Shallow layers benefit from the uniform attention for context lengths up to 25 words.

​

Toneva, Mariya, and Leila Wehbe. "Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)." Advances in neural information processing systems 32 (2019).

35

20-Jan-26

AAAI 2026: Brain-Inspired AI 2.0

Is the representation of semantic information different when reading vs listening?

Deniz, Fatma, Anwar O. Nunez-Elizalde, Alexander G. Huth, and Jack L. Gallant. "The representation of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modality." Journal of Neuroscience 39, no. 39 (2019): 7722-7736.

• Train on fMRI for 10 narrative stories (from The Moth Radio Hour) while participants listened to or read several hours
• Voxel-wise modeling with banded ridge regression
• During reading, each word was presented for a duration=duration of that word in the spoken story.
• Features
• 39 Motion-energy features using spatiotemporal Gabor pyramid
• 80 spectral audio features based on cochleogram.
• Word rate, phoneme rate, letter rate, word length variation per TR
• 39 phoneme frequency
• 26 letter frequency
• 56 syntactic binary features: 12 POS and 44 dependency
• 985 co-occurrence semantics

​

36

20-Jan-26

AAAI 2026: Brain-Inspired AI 2.0

Is the representation of semantic information different when reading vs listening?

Deniz, Fatma, Anwar O. Nunez-Elizalde, Alexander G. Huth, and Jack L. Gallant. "The representation of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modality." Journal of Neuroscience 39, no. 39 (2019): 7722-7736.

• Semantic tuning during listening and reading are highly correlated in most semantically selective regions of cortex
• Models estimated using one modality accurately predict voxel responses in the other modality.

37

20-Jan-26

AAAI 2026: Brain-Inspired AI 2.0

Do larger Transformer models lead to better brain‑encoding accuracy?

Schrimpf, Martin, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. "The neural architecture of language: Integrative modeling converges on predictive processing." Proceedings of the National Academy of Sciences 118, no. 45 (2021): e2105646118.

• Transformer models predict nearly 100% of explainable variance in neural responses to sentences.
• Larger models are better. GPT2-XL is the best.

​

38

20-Jan-26

AAAI 2026: Brain-Inspired AI 2.0

Does improved perf of Transformer models on NLP benchmarks translate to better brain‑encoding accuracy?

Schrimpf, Martin, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. "The neural architecture of language: Integrative modeling converges on predictive processing." Proceedings of the National Academy of Sciences 118, no. 45 (2021): e2105646118.

• Performance on GLUE tasks does not predict brain scores.

​

• Next-Word-Prediction Task Performance Selectively Predicts Brain Scores.

​

39

20-Jan-26

AAAI 2026: Brain-Inspired AI 2.0

Does improved perf of Transformer models on NLP benchmarks translate to better brain‑encoding accuracy?

Schrimpf, Martin, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. "The neural architecture of language: Integrative modeling converges on predictive processing." Proceedings of the National Academy of Sciences 118, no. 45 (2021): e2105646118.

• Perf on other GLUE tasks does not correlate with behavioral scores

• Behavioral scores (self-paced reading times), brain scores, and next-word-prediction task perf are pairwise correlated.

​

40

20-Jan-26

AAAI 2026: Brain-Inspired AI 2.0

Which NLP Tasks are the most Predictive of fMRI Brain Activity?

• Coreference resolution, NER, and shallow syntax parsing explain greater variance for the reading activity.
• For the listening activity, paraphrase generation, summarization, and natural language inference show better encoding performance.
• Tree derived from predicted brain responses vs tree based on Transformer encoder embeddings.

​

​

Oota, Subba Reddy, Jashn Arora, Veeral Agarwal, Mounika Marreddy, Manish Gupta, and Bapi Surampudi. "Neural language taskonomy: Which NLP tasks are the most predictive of fMRI brain activity?." In NAACL-HLT, pp. 3220-3237. 2022.

41

20-Jan-26

AAAI 2026: Brain-Inspired AI 2.0

Does the brain also perform next word prediction and is surprised when next word does not match the prediction?

• ECoG while listening to a 30-min podcast.
• Human brains and DLMs
• Both are engaged in continuous next-word prediction before word onset
• Both match their pre-onset predictions to the incoming word to calculate post-onset surprise
• Both rely on contextual embeddings to represent words in natural contexts.

​

Goldstein, Ariel, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A. Nastase et al. "Shared computational principles for language processing in humans and deep language models." Nature neuroscience 25, no. 3 (2022): 369-380.

42

20-Jan-26

AAAI 2026: Brain-Inspired AI 2.0

Does the brain also perform next word prediction and is surprised when next word does not match the prediction?

• GloVe yielded significant correlations with predicted neural responses to upcoming words up to 800 ms before word onset.
• Pre-onset activity associated with next-word prediction matches prediction content even when the prediction was incorrect.

​

Goldstein, Ariel, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A. Nastase et al. "Shared computational principles for language processing in humans and deep language models." Nature neuroscience 25, no. 3 (2022): 369-380.

• Post-onset activity matches content of the incoming word, even if it was unpredicted.
• Increase in encoding perf for surprising words compared to predicted words 400 ms after word onset.

43

20-Jan-26

AAAI 2026: Brain-Inspired AI 2.0

Does the brain depend on context for processing text?

• Match between confidence level and the accuracy level of GPT-2 and human predictions
• Under-confidence in their predictions and were above 95% correct when the probabilities were higher than 40%.
• Correlation between human and GPT2 word predictions improved as the contextual window increased.

​

Goldstein, Ariel, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A. Nastase et al. "Shared computational principles for language processing in humans and deep language models." Nature neuroscience 25, no. 3 (2022): 369-380.

• Contextual embeddings (GPT-2) > static embeddings (GloVe)
• Contextual embeddings (GPT-2) > concatenated GloVe embeddings for 10 prev words
• Avg context: Averaging all embeddings for each unique word (all occurrences of ‘monkey’) into 1 vector
• Scramble the embeddings across different occurrences of the same word in the story (switch embedding of ‘monkey’ in sentence 5 with that in sentence 50).

​

44

20-Jan-26

AAAI 2026: Brain-Inspired AI 2.0

Do speech models align with language and speech ROIs during speech production and comprehension?

• Extracted low-level acoustic, mid-level speech and contextual word embeddings from a multimodal speech-to-text model (Whisper).
• Sensory and motor regions better align with the model’s speech embeddings
• Higher-level language areas better align with the model’s language embeddings.

​

Goldstein, Ariel, Haocheng Wang, Leonard Niekerken, Mariano Schain, Zaid Zada, Bobbi Aubrey, Tom Sheffer et al. "A unified acoustic-to-speech-to-language embedding space captures the neural basis of natural language processing in everyday conversations." Nature human behaviour (2025): 1-15.

45

20-Jan-26

AAAI 2026: Brain-Inspired AI 2.0

Are both text and audio needed for speech alignment?

• Only text input (conversation transcripts) vs language receiving audio and text inputs (speech recordings and conversation transcripts)
• Language embeddings fused with auditory features outperform text-only language embeddings in predicting neural activity across multiple electrodes.
• Even though IFG is associated with linguistic processing, across multiple lags, the audio-fused language embeddings yield higher encoding performance during both production and comprehension.

​

Goldstein, Ariel, Haocheng Wang, Leonard Niekerken, Mariano Schain, Zaid Zada, Bobbi Aubrey, Tom Sheffer et al. "A unified acoustic-to-speech-to-language embedding space captures the neural basis of natural language processing in everyday conversations." Nature human behaviour (2025): 1-15.

46

20-Jan-26

AAAI 2026: Brain-Inspired AI 2.0

Which speech model aligns the best?

• Wav2Vec2.0, HuBERT, Data2Vec align well.
• Data2Vec aligns best with both language and auditory brain regions

​

Oota, Subba Reddy, Khushbu Pahwa, Mounika Marreddy, Manish Gupta, and Bapi S. Raju. "Neural architecture of speech." In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.

47

20-Jan-26

AAAI 2026: Brain-Inspired AI 2.0

Which Speech Tasks are the most Predictive of fMRI Brain Activity?

• 8 tasks from Speech processing Universal PERformance Benchmark (SUPERB): Phoneme Recognition (PR), Automatic Speech Recognition (ASR), Keyword Spotting (KS), Intent Classification (IC), Speaker Diarization (SD), Speaker Verification (SV), Speaker Identification (SID), and Emotion Recognition (ER)

​

​

​

Oota, Subba Reddy, Veeral Agarwal, Mounika Marreddy, Manish Gupta, and Raju Surampudi Bapi. "Speech taskonomy: Which speech tasks are the most predictive of fmri brain activity?." In INTERSPEECH 2023-24th INTERSPEECH Conference, pp. 5167-5171. 2023.

• ASR finetuning (middle layers) yields the best encoding performance for the whole brain, language and auditory regions.
• Finetuning on ER, SID and IC leads to the best alignment for the early auditory cortex.

48

20-Jan-26

AAAI 2026: Brain-Inspired AI 2.0

A big thank you!

49

20-Jan-26

Tutorial, Code and Material:

Material from AAAI 2026 Tutorial would be uploaded soon!

(Past): Deep Learning for Brain Encoding and Decoding, Cogsci-2022

https://tinyurl.com/DL4Brain

(Past): Language and the Brain: Deep Learning for Brain Encoding and Decoding, IJCNN 2023

https://tinyurl.com/DLBrainIJCNN2023

(Past): Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding, IJCAI 2023

https://tinyurl.com/DLBrainIJCAI2023

AAAI 2026: Brain-Inspired AI 2.0

Thanks!

• Questions
• subba-reddy.oota@inria.fr
• tanchak@iitd.ac.in
• gmanish@microsoft.com
• raju.bapi@iiit.ac.in
• Connect with us:
• https://www.linkedin.com/in/subba-reddy-oota-11a91254/
• https://tanmoychak.com/
• http://aka.ms/manishgupta, https://sites.google.com/view/manishg/
• https://sites.google.com/view/bccl-iiith/home

​

​

​

​

​

AAAI 2026: Brain-Inspired AI 2.0