1 of 42

Foundation models of the

human mind

Lecture 2: Pretrained models & human brain and behavior

Acknowledgment: first draft of slides prepared with Claude

2 of 42

Are current AI models foundation models of the mind?

2

Big improvement over previous computational models

  • Much better generalization to new stimuli
  • Strong prediction of human brain activity and behavior

But pieces are still missing

  • Predictive functions still need lots of per-participant data
  • Only tailored to very specific human data (language generation and preferences)

​

3 of 42

Limited cognitive data for training such a model from scratch

3

~1,000h

Biggest foundation model for fMRI (TribeV2) trained on ~1000h of data

Current state

The next 5 years!

+15,000h

Meta’s Digital Brain Project

​

Stanford’s Digital Brain Project

Still, much smaller scale than data AI models are trained on

4 of 42

Core idea: leverage pretrained AI models + tune with diverse human data

4

Pretrained AI model

General representations from web-scale data

Tune with diverse human data

Brain recordings and/or behavior

Improved foundation model of the human mind

Better aligned with humans

across tasks

Today

Frozen models: human data only evaluates the model or fits a small readout

Next time: Background Lecture 2

Tuned models: human data changes the model itself

5 of 42

Today’s outline

  • Human data:
    • Experimental paradigms: controlled vs naturalistic
    • Brain imaging modalities

​

  • Alignment of pretrained AI models with human data

​

  • How to evaluate alignment of AI models and human brain data
    • Encoding, decoding, RSA

​

​

6 of 42

Today’s outline

  • Human data:
    • Experimental paradigms: controlled vs naturalistic
    • Brain imaging modalities & behavioral measures

​

  • Alignment of pretrained AI models with human data

​

  • How to evaluate alignment of AI models and human brain data
    • Encoding, decoding, RSA

​

​

7 of 42

AI models need stimulus-computable input

7

Easy to feed to a model

  • Images and movies
  • Speech audio
  • Text

Hard to feed to a model

  • Memory
  • Mental imagery
  • Mood and internal states

Both start from a pretrained AI model

Focus specifically on vision and language modalities in the human

8 of 42

Two types of human studies: naturalistic vs controlled

8

Naturalistic

  • Vision stimuli: movies, images of natural scenes
  • Language stimuli: books, audiobooks
  • Behavioral tasks: eye movements, free recall

​

​

Controlled

  • Vision stimuli: object images and categories
  • Language: isolated words, selected sentences
  • Behavioral tasks: similarity judgments, categorization, choices

Rich and more ecologically valid

Many samples

Correlated features and autocorrelation

Clean contrasts

Easy to interpret

Little data per condition

A caution especially for naturalistic data

A model may predict the brain by capturing word frequency or acoustics rather than meaning. Control analyses are essential.

9 of 42

The brain data recording modality also matters

9

What can be predicted

Where, when, or at what cell-level resolution

How much data

Hours per person, or thousands of neurons

How noisy

Signal-to-noise differs by orders of magnitude

10 of 42

Brain recording modalities at a glance

10

Modality

Space / time

Invasive?

Typical data scale

Typical use with models

fMRI

mm / seconds

No

Large, dense (NSD, story and movie sets)

Whole-brain encoding, decoding, RSA

EEG

Coarse / milliseconds

No

Many participants, low SNR

Timing of representations

MEG

Coarse / milliseconds

No

Few participants, lower SNR than fMRI but higher than EEG

Timing of representations

ECoG / sEEG

mm to cm / milliseconds

Yes (patients)

Small N, high SNR

Fine-grained language and speech

Single units, calcium imaging

Cells / ms to s

Yes (mostly animals)

Thousands of neurons

Mechanistic comparison

11 of 42

fMRI: where, with a slow signal

11

  • BOLD signal: an indirect, blood-oxygen-based measure
  • Whole-brain coverage at roughly mm resolution
  • Model the lag with delayed copies of the features (FIR)
  • Large datasets: NSD, THINGS, naturalistic story and movie sets

~5 s

Hemodynamic response peak

1-2 s

Typical sampling interval

mm

Voxel-scale spatial resolution

12 of 42

fMRI: reading a predictive-performance figure

12

How to read it

  • Above: colors = region involved in language or/and vision
  • Below:
    • encoding score = correlation between predicted and measured fMRI
    • The peak moves with the stimulus: speech to temporal cortex, video to visual cortex

Nice 3D brain schematic predictive performance visualization: https://ai.meta.com/blog/tribe-v2-brain-predictive-foundation-model/

Fig. 2, d'Ascoli et al. 2026 (TRIBE v2)

13 of 42

MEG / EEG: when

13

  • Millisecond timing, coarse spatial localization
  • Lower signal-to-noise per trial, many participants possible in EEG
  • Ask: when do specific features appear in the brain signal?

​

ms

Temporal resolution

cm

Coarse spatial localization

Low

Signal-to-noise per trial

14 of 42

MEG / EEG: reading a predictive-performance figure

14

How to read

  • Time axis: how does the predictive score change after stimulus onset?
  • Sensors or sources: spatial detail is coarse

​

Toneva et al. 2022, Nature Comp Science

Proportion of sensors sig. predicted

15 of 42

ECoG and sEEG: invasive human recordings

15

Patients

Recorded during clinical monitoring, for example for epilepsy

High quality signal

High SNR and millisecond timing

Limited coverage

Electrode locations set by clinical need, mm to cm scale

Small samples

Few participants, but strong effects in speech and language

16 of 42

ECoG / sEEG: reading a predictive-performance figure

16

How to read:

  • Each marker is an electrode, colored by predictive encoding score
  • Blank areas mean no coverage
  • Coverage is set by clinical need and differs per patient
  • Lag plots: score as a function of time around word onset

Goldstein et al. 2022, Nature Neuro

17 of 42

What is a good predictive score?

17

  • Noise ceiling: the reproducible variance, estimated from repeated measurements
  • Report accuracy relative to the ceiling
  • Otherwise predictive scores are not comparable across datasets or modalities

18 of 42

Today’s outline

  • Human data:
    • Experimental paradigms: controlled vs naturalistic
    • Brain imaging modalities & behavioral measures

​

  • Alignment of pretrained AI models with human data

​

  • How to evaluate alignment of AI models and human brain data
    • Encoding, decoding, RSA

​

​

19 of 42

Human behavior: can directly prompt model

Binz & Schulz 2023

GPT-3 on classic cognitive tasks: human-like on some, not on others

Silicon samples

Argyle et al. 2023: model responses as stand-ins for survey samples

LLMs as participants

Dillion et al. 2023: can models replace participants?

Handle with care

Results are sensitive to prompt wording and format.

20 of 42

Frozen vision models and human behavior

20

Object recognition behavior

Rajalingham et al. 2018: deep networks predict which objects humans confuse and recognize.

Similarity and odd-one-out

Embeddings predict human similarity judgments (THINGS; Muttenthaler et al. 2023).

21 of 42

Where frozen models fall short

21

Systematic deviations

Models differ from humans in consistent ways

Prompt sensitivity

Small wording and format changes alter results

Training-data contamination

Classic tasks may be in the pretraining data

Narrow benchmarks

Mostly WEIRD samples and lab tasks

22 of 42

Estimating alignment of AI models with the brain

22

Same stimuli

Images, speech, text

Model representations

Brain activity

Compare

How well does one explain the other?

A match is evidence of shared computational structure, not of identical mechanism

23 of 42

The recipe: pretrain, extract, align

23

1 Pretrain

Classification, self-supervised, CLIP, next-token

2 Extract

Activations for the stimuli humans saw

3 Align

Regression or similarity analysis

Human datasets

Hours per person for brain data, up to millions of trials for behavior

Pretraining data

Web-scale: orders of magnitude larger

So we borrow

Reuse the learned representations and fit fewer parameters

24 of 42

Vision: hierarchical alignment with task-optimized models

24

Yamins et al. 2014; Khaligh-Razavi & Kriegeskorte 2014

  • Trained on object recognition, no brain data used
  • Predict V4 and IT better than earlier models
  • Layer hierarchy maps onto cortical hierarchy

25 of 42

Vision: modern updates

25

Self-supervised and CLIP-style

Fit visual cortex as well as supervised models

Conwell et al. 2024

Training data and objective matter more than architecture

Doerig et al. 2025

LLM embeddings of image captions predict high-level visual cortex

26 of 42

Language: from word embeddings to LLMs

26

2008

Word-level features

Mitchell et al. 2008: word features predict fMRI patterns

2012-2016

Semantic maps

Huth et al. 2012, 2016: word embeddings map meaning across cortex

2018-2021

Transformers

Toneva & Wehbe 2019: predict fMRI and MEG from book; Schrimpf et al. 2021: predict fMRI and ECoG; next-word accuracy tracks brain predictivity

27 of 42

Language: prediction and scaling

27

Prediction

Caucheteux & King 2022; Goldstein et al. 2022: information about upcoming words appears in neural signals

Scaling

Antonello et al. 2023: encoding performance grows with model size, then saturates

Caveat

Part of the fit may come from shared linguistic statistics, not a shared mechanism

28 of 42

Today’s outline

  • Human data:
    • Experimental paradigms: controlled vs naturalistic
    • Brain imaging modalities & behavioral measures

​

  • Alignment of pretrained AI models with human data

​

  • How to evaluate alignment of AI models and human brain data
    • Encoding, decoding, RSA

​

​

29 of 42

Three main methods for relating AI models to human data

29

Encoding

  • Model → brain
  • Can model features predict brain responses?

Decoding

  • Brain → stimulus or model features
  • Can we read content out of the brain?

RSA

  • Geometry vs geometry
  • Do model and brain treat stimuli as similar in the same way?

Running example: natural image viewing in fMRI

30 of 42

Encoding models: the pipeline

30

Stimulus

Image the person saw

Model features

One layer's activations

Ridge regression

Fit on training stimuli

Predicted response

Per voxel or electrode

Evaluate on held-out stimuli

Correlate predicted and measured responses on images never used for fitting.

Compare with the noise ceiling

Report how much of the explainable variance the model captures.

31 of 42

Encoding models: practical details

31

Layer choice

Select by cross-validation, never on test data

Regularization

Ridge strength chosen by cross-validation

Hemodynamic delays

Delayed copies of features for fMRI

Banded ridge

Combine feature spaces, each with its own regularization

Variance partitioning

Ask what one feature set adds over another

32 of 42

Encoding: what you can and cannot conclude

32

You can conclude

  • Which regions carry information that is linearly accessible from the model
  • Which layer best matches which region

You cannot conclude

  • That the brain computes like the model
  • That the geometry of the two spaces matches

A linear map picks whichever dimensions help, so a good fit means the information is present

33 of 42

Decoding: reading out from the brain

33

Brain activity

Fitted mapping

Model embedding or label

Classification

Which category was seen?

Identification

Which of these images was seen?

Reconstruction

Generate the image or text itself

34 of 42

Decoding in vision: reconstruction

34

fMRI response

CLIP-like embedding

Latent diffusion model

Reconstructed image

Examples

  • Takagi & Nishimoto 2023
  • MindEye (Scotti et al. 2023)
  • MinD-Vis (Chen et al. 2023)

Read with care

The diffusion prior does much of the work: part of what looks like mind reading is the generative model's knowledge of natural images.

35 of 42

Decoding in language

35

Language model

Proposes candidate text

Encoding model

Predicts brain response to each candidate

Compare

Match to recorded fMRI

Keep the best

Extend and repeat

Tang et al. 2023

Continuous language from fMRI. Recovers the gist of the text, not word-for-word output.

Constraints and ethics

Needs long subject-specific training and a cooperative participant. It fuels the mental privacy debate.

36 of 42

RSA: comparing geometry

36

  • Build a dissimilarity matrix (RDM) from model features
  • Build another from brain responses, over the same stimuli
  • Compare the two RDMs, usually with Spearman correlation
  • No fitted mapping between spaces

ρ

Model RDM

Brain RDM

Illustrative: 6 stimuli, darker = more dissimilar

37 of 42

RSA vs encoding

37

RSA

  • No fitted weights
  • Compares overall geometry
  • Sensitive to dominant dimensions

Encoding

  • Reweights model dimensions
  • More flexible, usually higher scores
  • Can succeed where RSA fails

They can disagree

Strong model dimensions that the brain lacks lower the RSA score, while a linear map can ignore them and keep encoding high.

38 of 42

Choosing the method

38

Method

Question

Fitted mapping?

Output

Encoding

Can model features predict brain responses?

Yes

Prediction accuracy per voxel or electrode

Decoding

Can we read stimulus content out of the brain?

Yes

Classified or reconstructed stimuli

RSA

Do model and brain share representational geometry?

No

Similarity of RDMs

39 of 42

Important considerations

39

Train/test leakage

Never evaluate on stimuli the readout saw

Low-level confounds

Pixels, word frequency or acoustics can drive the fit

Autocorrelation

Neighboring samples are not independent: split in blocks

Ignoring noise ceilings

Report accuracy relative to the ceiling

High score = brain?

Correspondence is not mechanism

40 of 42

Today’s outline

  • Human data:
    • Experimental paradigms: controlled vs naturalistic
    • Brain imaging modalities

​

  • Alignment of pretrained AI models with human data

​

  • How to evaluate alignment of AI models and human brain data
    • Encoding, decoding, RSA

​

​

41 of 42

How much human data is used in the overall model?

41

TODAY: frozen backbone

LECTURE 3: backbone is tuned

RSA

Compare geometry, no fitting

Linear readout

Encoding and decoding

Adapters / LoRA

A few added parameters

Full fine-tuning

All weights updated

Trained from scratch

Only human data

More human data →

42 of 42

Next class on Nov 19 in a different room -- 105

  • No class next week Nov 12
  • Register for the “examination” in LSF by Nov 12
  • Debates: will assign everyone to a debate and side randomly on Nov 12
  • Dec 3: Project Q&A after class, 12-1pm?
  • Questions?