The neural code�Lessons from machine learning

Kenneth D Harris, UCL

Multiple linear regression

Too many predictors

Overfitting = large weight vectors

Example

Ridge regression introduces a bias

Equivariance

Equivariance of ridge and linear regression

Delta rule

• Learning rule for 1-layer neural networks
• “Hebbian” coincidence detector of input with error

Delta rule vs. linear regression

There are problems linear regression can’t solve

​

• Not all patterns can be linearly separated

Solution: non-linear hidden units

Input

Representation

Output

“Codon” theory

Dense input

Sparse representation

Dense input

Sparse representation

“Untangling hypothesis”

• Cortex forms a high-dimensional representation of its inputs
• This allows nonlinear discriminations to be done with linear readout
• The cortical code could be – but does not need to be sparse

“No free lunch theorem”

• There is no representation that is good for all problems
• “If you can learn anything, you can’t learn anything”

​

• Neural representations have evolved to provide an “inductive bias” that lets animals quickly learn tasks that they will likely encounter
• This makes them bad at other tasks

​

• E.g. humans find it hard to distinguish two white noise images (different in every pixel)
• And easy to distinguish male and female faces (subtle differences in a few pixels)

​

• All human languages share grammatical features
• LLMs can learn artificial languages that do not share these features

Primate IT cortex visual code predicts human performance

• Record primate IT responses to images
• Including variations in pose, position, scale, etc.

​

​

​

​

• Train a linear decoder to distinguish image pairs

​

• Its performance correlates with human psychophysics

​

How to characterize a code

Equivalent to

Kernel matrix vs covariance matrix

Kernel matrix vs. Kernel function

Kernel eigenfunctions

Relation between eigenvalues

• As the number of neurons and stimuli increases:

​

The sample kernel matrix eigenvalues converge to �the kernel function eigenvalues.

​

• If we measure responses to enough stimuli, we can estimate kernel function eigenvalues

​

• Like PCA but for all stimuli we could have shown, not just those we did

​

V. Koltchinskii & E. Giné, Berrnoulli, 2000

Higher eigenfunctions encode finer stimulus features

https://github.com/MouseLand/rastermap

What the eigenspectrum means

fractal

discontinuous

An experimental prediction

Low-dimensional inputs

Low dimensional stimuli:

Summary

• Cortex may form high-dimensional representations of stimuli to allow linear readout

​

• Representational geometry is summarized by the kernel function: the dot-product of representations of every stimulus pair.

​

• For rotation-equivariant readouts, this is the only thing that matters

​

• Kernel eigenvalues tell you how much weight the representation gives to fine details vs. big picture

​

• V1 eigenvalues decay just slowly enough to not be pathological