Geometry and Latent Signal Representations in Machine Learning

Daniel D. Lee

Outline

• How to understand empirical successes of large and deep neural networks?
• Neural representations and manifolds
• Statistical theory of manifold geometry for latent signals
• Manifolds and generative models

​

Engineering applications

• Development of autonomous robots and AI devices

Samsung AI Centers

DARPA Challenges

RoboCup

Power of scaling

Tan and Le (2019)

Hoffmann, et. al. (2022)

• Moore’s law and Internet-scale data fueling large model performance improvements

Malthus’ Law

• Limit to the scale of human created content

AI echo chambers

• Difficulties in proper segregation between training, validation and test evaluation metrics

Physics and symmetries

• Fourier modes are the eigenfunctions for translationally invariant signals

Brain Prize 2024

• Three theoretical physicists awarded top prize in computational neuroscience

Ring attractor neural network

• Fourier modes used to construct a nonlinear recurrent neural network (Ben-Yishai, et. al., 1995)

ReLU

Ring attractor geometry

• Continuous, non-linear, low-dimensional manifold (ring) of stable states in neural activity space

Neural activity space

Biological head direction system

• Head direction (compass) neurons recorded in the ellipsoidal body of the fly brain

Jayaraman (2017)

Larger and deeper networks

• Success of more flexible models with increasing number of parameters

Conventional filterbank approach

• Failure to model deeper layers as filters with receptive fields and response kernels

Hubel and Wiesel (1963)

Manifold hypothesis

• Neural manifold representations become “untangled” in deeper layers of processing

DiCarlo and

Cox (2007)

Manifold geometry and perception

SueYeon Chung

(NYU/Flatiron)

​

Haim Sompolinsky

(Hebrew U./Harvard)

Uri Cohen

(U. Cambridge)

“Classification and geometry of general perceptual manifolds.” Physical Review X 8.3 (2018).

“Separability and geometry of object manifolds in deep neural networks.” Nature Comm. 11.1 (2020).

Signal data

🐶

🐱

Vector

Representation

• High-dimensional signals for multi-modal data inputs

Perceptron model

Frank Rosenblatt (1958)

• Classic biologically-inspired model of a logic neuron

Perceptron geometry

• Weight vector and decision surface in embedding space

Linear separability

Training examples:

Linear classifier:

• Output layer of neural networks is a simple linear classifier

Cover counting theorem

• Dichotomies of points in general position

Number of linearly separable binary labellings�of P points in N dimensions:

Thomas Cover

(1965)

Recursion relation

• Solve recursion relation between number of separable dichotomies with one additional point

P+1 point

Perceptron capacity

• Phase transition in capacity of perceptron as N goes to infinity

VC dim

α_c=2

Margin

• Large margin solution is robust to noise for better generalization error

Many possible solutions

Maximum margin

Gardner theory

• Compute volume of hypothesis space

Elizabeth Gardner

(1988)

Volume of hypothesis space:

Replica trick

• Reduction to Gaussian integrals and analytic continuation

Entropy is expected log volume:

Gardner results

• Theoretical prediction for number of points at capacity, and fraction in interior or on the margin

Capacity as a function of margin:

Separable region

Data manifolds

• Variability in pose, lighting, and expression give rise to low dimensional manifold structures

Lawrence Saul

Classifying manifolds

• Conventional approaches use data augmentation to increase number of training points by sampling manifolds

Dichotomies of P manifolds�in N dimensions:

Manifold model

• Mathematical description of the manifold�in terms of its convex hull

Geometrical description

Manifold capacity limits

• Critical number of manifolds can vary greatly as size and shape are changed

Small (point limit):

Large (full hyperplanes):

Random point cloud (m points):

Manifold capacity formula

• Prediction for maximum number of separable manifolds in terms of the margin

Critical number

of separable manifolds:

Effective geometry

• Classification capacity of manifolds can be described by an effective radius and dimension

Interior balls

Embedded balls

Touching balls

Touching balls

Manifold radius:

Manifold dimension:

Support manifolds

• Taxonomy of manifolds that are embedded in the margin planes

Margin planes

Deep network analysis

• Analyzing the representations in different layers of a deep networks

VGG-16

ResNet

Input manifolds

• Analysis of two types of image manifolds: continuous affine warped manifold and point cloud convex hulls from ImageNet

Affine warp manifold

ImageNet point cloud

Classification capacity

• Capacity can be used to compare between layers as well as between different architectures

Manifold geometrical measures

• Track changes to effective manifold radius and dimension through network layers

One-shot learning

• Generalization error of one or few-shot learning predicted by manifold geometry

Sorcher, et. al.

(2021)

Neural data analysis

• Dynamical changes to the geometry of neural activity manifolds in the brain

Chou, et. al.

(preprint)

Froudarakis, et. al.

(preprint)

Generative diffusion models

• Generating probabilistic samples from data manifolds

Diffusion models

Ho, et. al. (2020)

• Forward diffusion process iteratively adds Gaussian noise to data samples

Diffusion dynamics

Stochastic differential equation:

Fokker-Planck equation:

Reverse diffusion drift score function:

Manifold diffusion model

Chung, et. al. (2022)

• Novel score function dynamics with iterative manifold projections

Summary

• Understanding structure of data signals and representations in machine learning
• Geometrical measures of manifolds for classification performance
• Manifold analysis of neural brain data and machine learning networks
• Incorporating geometry can lead to more efficient algorithms

Messy embodied AI

Very large continuous state and action spaces, unknown and uncertainty in dynamics, nonstationary environments

Still much to learn