Foundation models of the
human mind
Lecture 2: Pretrained models & human brain and behavior
Acknowledgment: first draft of slides prepared with Claude
Are current AI models foundation models of the mind?
2
Big improvement over previous computational models
But pieces are still missing
Limited cognitive data for training such a model from scratch
3
~1,000h
Biggest foundation model for fMRI (TribeV2) trained on ~1000h of data
Current state
The next 5 years!
+15,000h
Meta’s Digital Brain Project
Stanford’s Digital Brain Project
Still, much smaller scale than data AI models are trained on
Core idea: leverage pretrained AI models + tune with diverse human data
4
Pretrained AI model
General representations from web-scale data
Tune with diverse human data
Brain recordings and/or behavior
Improved foundation model of the human mind
Better aligned with humans
across tasks
Today
Frozen models: human data only evaluates the model or fits a small readout
Next time: Background Lecture 2
Tuned models: human data changes the model itself
Today’s outline
Today’s outline
AI models need stimulus-computable input
7
Easy to feed to a model
Hard to feed to a model
Both start from a pretrained AI model
Focus specifically on vision and language modalities in the human
Two types of human studies: naturalistic vs controlled
8
Naturalistic
Controlled
Rich and more ecologically valid
Many samples
Correlated features and autocorrelation
Clean contrasts
Easy to interpret
Little data per condition
A caution especially for naturalistic data
A model may predict the brain by capturing word frequency or acoustics rather than meaning. Control analyses are essential.
The brain data recording modality also matters
9
What can be predicted
Where, when, or at what cell-level resolution
How much data
Hours per person, or thousands of neurons
How noisy
Signal-to-noise differs by orders of magnitude
Brain recording modalities at a glance
10
Modality | Space / time | Invasive? | Typical data scale | Typical use with models |
fMRI | mm / seconds | No | Large, dense (NSD, story and movie sets) | Whole-brain encoding, decoding, RSA |
EEG | Coarse / milliseconds | No | Many participants, low SNR | Timing of representations |
MEG | Coarse / milliseconds | No | Few participants, lower SNR than fMRI but higher than EEG | Timing of representations |
ECoG / sEEG | mm to cm / milliseconds | Yes (patients) | Small N, high SNR | Fine-grained language and speech |
Single units, calcium imaging | Cells / ms to s | Yes (mostly animals) | Thousands of neurons | Mechanistic comparison |
fMRI: where, with a slow signal
11
~5 s
Hemodynamic response peak
1-2 s
Typical sampling interval
mm
Voxel-scale spatial resolution
fMRI: reading a predictive-performance figure
12
How to read it
Nice 3D brain schematic predictive performance visualization: https://ai.meta.com/blog/tribe-v2-brain-predictive-foundation-model/
Fig. 2, d'Ascoli et al. 2026 (TRIBE v2)
MEG / EEG: when
13
ms
Temporal resolution
cm
Coarse spatial localization
Low
Signal-to-noise per trial
MEG / EEG: reading a predictive-performance figure
14
How to read
Toneva et al. 2022, Nature Comp Science
Proportion of sensors sig. predicted
ECoG and sEEG: invasive human recordings
15
Patients
Recorded during clinical monitoring, for example for epilepsy
High quality signal
High SNR and millisecond timing
Limited coverage
Electrode locations set by clinical need, mm to cm scale
Small samples
Few participants, but strong effects in speech and language
ECoG / sEEG: reading a predictive-performance figure
16
How to read:
Goldstein et al. 2022, Nature Neuro
What is a good predictive score?
17
Today’s outline
Human behavior: can directly prompt model
Binz & Schulz 2023
GPT-3 on classic cognitive tasks: human-like on some, not on others
Silicon samples
Argyle et al. 2023: model responses as stand-ins for survey samples
LLMs as participants
Dillion et al. 2023: can models replace participants?
Handle with care
Results are sensitive to prompt wording and format.
Frozen vision models and human behavior
20
Object recognition behavior
Rajalingham et al. 2018: deep networks predict which objects humans confuse and recognize.
Similarity and odd-one-out
Embeddings predict human similarity judgments (THINGS; Muttenthaler et al. 2023).
Where frozen models fall short
21
Systematic deviations
Models differ from humans in consistent ways
Prompt sensitivity
Small wording and format changes alter results
Training-data contamination
Classic tasks may be in the pretraining data
Narrow benchmarks
Mostly WEIRD samples and lab tasks
Estimating alignment of AI models with the brain
22
Same stimuli
Images, speech, text
Model representations
Brain activity
Compare
How well does one explain the other?
A match is evidence of shared computational structure, not of identical mechanism
The recipe: pretrain, extract, align
23
1 Pretrain
Classification, self-supervised, CLIP, next-token
2 Extract
Activations for the stimuli humans saw
3 Align
Regression or similarity analysis
Human datasets
Hours per person for brain data, up to millions of trials for behavior
Pretraining data
Web-scale: orders of magnitude larger
So we borrow
Reuse the learned representations and fit fewer parameters
Vision: hierarchical alignment with task-optimized models
24
Yamins et al. 2014; Khaligh-Razavi & Kriegeskorte 2014
Vision: modern updates
25
Self-supervised and CLIP-style
Fit visual cortex as well as supervised models
Conwell et al. 2024
Training data and objective matter more than architecture
Doerig et al. 2025
LLM embeddings of image captions predict high-level visual cortex
Language: from word embeddings to LLMs
26
2008
Word-level features
Mitchell et al. 2008: word features predict fMRI patterns
2012-2016
Semantic maps
Huth et al. 2012, 2016: word embeddings map meaning across cortex
2018-2021
Transformers
Toneva & Wehbe 2019: predict fMRI and MEG from book; Schrimpf et al. 2021: predict fMRI and ECoG; next-word accuracy tracks brain predictivity
Language: prediction and scaling
27
Prediction
Caucheteux & King 2022; Goldstein et al. 2022: information about upcoming words appears in neural signals
Scaling
Antonello et al. 2023: encoding performance grows with model size, then saturates
Caveat
Part of the fit may come from shared linguistic statistics, not a shared mechanism
Today’s outline
Three main methods for relating AI models to human data
29
Encoding
Decoding
RSA
Running example: natural image viewing in fMRI
Encoding models: the pipeline
30
Stimulus
Image the person saw
Model features
One layer's activations
Ridge regression
Fit on training stimuli
Predicted response
Per voxel or electrode
Evaluate on held-out stimuli
Correlate predicted and measured responses on images never used for fitting.
Compare with the noise ceiling
Report how much of the explainable variance the model captures.
Encoding models: practical details
31
Layer choice
Select by cross-validation, never on test data
Regularization
Ridge strength chosen by cross-validation
Hemodynamic delays
Delayed copies of features for fMRI
Banded ridge
Combine feature spaces, each with its own regularization
Variance partitioning
Ask what one feature set adds over another
Encoding: what you can and cannot conclude
32
You can conclude
You cannot conclude
A linear map picks whichever dimensions help, so a good fit means the information is present
Decoding: reading out from the brain
33
Brain activity
Fitted mapping
Model embedding or label
Classification
Which category was seen?
Identification
Which of these images was seen?
Reconstruction
Generate the image or text itself
Decoding in vision: reconstruction
34
fMRI response
CLIP-like embedding
Latent diffusion model
Reconstructed image
Examples
Read with care
The diffusion prior does much of the work: part of what looks like mind reading is the generative model's knowledge of natural images.
Decoding in language
35
Language model
Proposes candidate text
Encoding model
Predicts brain response to each candidate
Compare
Match to recorded fMRI
Keep the best
Extend and repeat
Tang et al. 2023
Continuous language from fMRI. Recovers the gist of the text, not word-for-word output.
Constraints and ethics
Needs long subject-specific training and a cooperative participant. It fuels the mental privacy debate.
RSA: comparing geometry
36
ρ
Model RDM
Brain RDM
Illustrative: 6 stimuli, darker = more dissimilar
RSA vs encoding
37
RSA
Encoding
They can disagree
Strong model dimensions that the brain lacks lower the RSA score, while a linear map can ignore them and keep encoding high.
Choosing the method
38
Method | Question | Fitted mapping? | Output |
Encoding | Can model features predict brain responses? | Yes | Prediction accuracy per voxel or electrode |
Decoding | Can we read stimulus content out of the brain? | Yes | Classified or reconstructed stimuli |
RSA | Do model and brain share representational geometry? | No | Similarity of RDMs |
Important considerations
39
Train/test leakage
Never evaluate on stimuli the readout saw
Low-level confounds
Pixels, word frequency or acoustics can drive the fit
Autocorrelation
Neighboring samples are not independent: split in blocks
Ignoring noise ceilings
Report accuracy relative to the ceiling
High score = brain?
Correspondence is not mechanism
Today’s outline
How much human data is used in the overall model?
41
TODAY: frozen backbone
LECTURE 3: backbone is tuned
RSA
Compare geometry, no fitting
Linear readout
Encoding and decoding
Adapters / LoRA
A few added parameters
Full fine-tuning
All weights updated
Trained from scratch
Only human data
More human data →
Next class on Nov 19 in a different room -- 105