NeurIPS 2023

Trends in AI

Nikolaos Vasiloglou

VP of Research-ML

1

1

A few words

• This review was based on:
• Tutorials and Workshops
• Keynotes and best papers
• Did not go through the total of ~7.5K papers
• Tried to dig into references from the Tutorials and Workshops that come from other conferences as well
• Compiled a comprehensive list of links to papers and talks for further reading
• While papers are free, talks will become free for the public soon (You can always “buy” a registration and view them)
• By attending one of the three ICLR/ICML/NeurIPS you get a good view of AI in the current year

​

2

2

In a nutshell (1)

• LLMs are everywhere but not for everyone yet
• Democratization of LLMs
• Free models (LLAMA 2)
• The weights are public but not the training code
• Open Source Models (LLM360)
• Training data, code, checkpoints are public
• Fully reproducible
• Open Infrastructure (CollosalAI)
• All the distribution infrastructure you need to train a huge model
• Full stack language (Mojo)
• Unifying the software layers for better LLM coding
• Composable LLMS
• Modularizing LLMs by combining smaller ones

3

3

In a nutshell (2)

• LLMs for Math Reasoning
• The return of theorem provers and their applications on other domains
• Theorem Provers might be a great companion for LLMs
• Are we running out of data for LLMs?
• The unexpected value of simulators
• Relational Tables and Language Models
• Predictive operations with LLMs
• Hopfield Networks are back
• Creating hope for better theoretical understanding of Attention
• There was no tutorial/workshop about GNNs, only 42 papers/4000 in the main conference. Are transformers winning over GNNs?

4

4

AI futurism

• New Frontiers of AI for Drug Discovery and Development
• Machine Learning with New Compute Paradigms

​

​

5

5

Other NeurIPS reviews

6

https://www.latent.space/p/neurips-2023-papers

6

Understanding LLMs

7

7

Optimizing attention

Systems for Foundation Models, and Foundation Models for Systems. [pptx] Christopher Ré

​

​

8

8

Test of time (Word2vec)

Distributed Representations of Words and Phrases and their Compositionality

Tomas Mikolov* · Ilya Sutskever* · Kai Chen · Greg Corrado · Jeff Dean

​

​

​

​

9

*Absent in the ceremony

9

What we have learned so far

• The birth of self-supervision
• Word2vec -> Elmo -> Transformer (BERT)
• Context free embeddings -> Context Aware Embeddings
• The return of Asynchronous Training

10

10

An ontology of word embeddings (Before GPT)

11

https://spotintelligence.com/2023/12/26/embeddings-from-language-models-elmo/

Something is missing

11

The revolution of context

12

https://spotintelligence.com/2023/12/26/embeddings-from-language-models-elmo/

12

The Language Model is the Embedding

• In word2vec the word-vector is the embedding
• Semantic Reasoning with Vector arithmetic

10 years later

“Efficiently Tuned Parameters are Task Embedding,

“Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models,

“TASK2VEC: Task Embedding for Meta-Learning”

​

13

Jump here if you want more!

13

The unreasonable behavior of LLMs

What is going on as we scale from thousand to million to billion to trillion parameters

14

14

Do LLMs have emergent properties?

Are Emergent Abilities of Large Language Models a Mirage?

​

15

15

Are we running

out of oil?

Sustainability analysis of data

16

16

Managing data repetitions and parameter budget

• Scaling Data-Constrained Language Models (Outstanding Runner up) [paper]
• Will we run out of data? An analysis of the limits of scaling datasets in Machine Learning
• High Quality data (Books and papers 1TB/16TB) will be exhausted by 2024
• Code data 14TB (unclear on the usefulness)
• Repeating data gives you a small edge
• The best thing to do is replace some of the data with code and then repeat

​

17

17

Training data trend

18

18

Training LLMs on data budget

19

19

LLMs as a World Model

Tutorial Language Models Meet World Models [slides]

20

20

The rise of simulators

21

21

Train LLMs with Simulation data!

22

22

LLMs not good enough for Common Sense

23

23

Literature

24

On the Planning Abilities of Large Language Models : A Critical Investigation

Reasoning with Language Model is Planning with World Model

24

LLM not good enough for Social Tasks

25

25

Literature

26

• MuJoCo — Advanced Physics Simulation
• iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks
• AI2-THOR: An Interactive 3D Environment for Visual AI
• Habitat 2.0: Training Home Assistants to Rearrange their Habitat
• ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation
• Learning Interactive Real-World Simulators
• SceneScript: Reconstructing Scenes With An Autoregressive Structured Language Model

​

​

26

Literature

27

• Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World
• Learning rigid dynamics with face interaction graph networks
• Graph networks as learnable physics engines for inference and control

27

A deep dive into the physics simulators

Recent Advances

28

28

MuJoCo: A physics engine for model-based control

Multi-Joint dynamics with Contact

• one order of magnitude is due to faster computation,
• one order of magnitude is due to parallel processing which fully utilizes all available processors
• one order of magnitude is due to higher accuracy and stability allowing larger time-steps

29

29

iGibson 2.0

Object-Centric Simulation for Robot Learning of Everyday Household Tasks

• supports object states,
• temperature, wetness level, cleanliness level, and toggled and sliced states,
• implements a set of predicate logic functions that map the simulator states to logic states like Cooked or Soaked.
• given a logic state, iGibson 2.0 can sample valid physical states that satisfy it.
• can generate potentially infinite instances of tasks with minimal effort
• includes a virtual reality (VR) interface to immerse humans in its scenes to collect demonstrations.

30

30

iGibson 2.0

31

31

Habitat 2.0

Training Home Assistants to Rearrange their Habitat

​

• ReplicaCAD: an artist-authored, annotated, reconfigurable 3D dataset of apartments with articulated objects (e.g. cabinets and drawers that can open/close)
• H2.0: a high-performance physics-enabled 3D simulator with speeds, representing 100× speed-ups over prior work
• Home Assistant Benchmark (HAB): a suite of common tasks for assistive robots (tidy the house, prepare groceries, set the table)

32

32

AI2-THOR

An Interactive 3D Environment for Visual AI

• The House Of inteRactions (THOR)
• Near photo-realistic 3D indoor scenes
• AI agents can navigate in the scenes and interact with objects
• Enables research in many different domains
• Deep reinforcement learning
• imitation learning, learning by interaction
• Planning,
• visual question answering
• Learning models of cognition.

​

33

33

AI2-THOR

34

34

A big library

35

35

Habitat 2.0

36

36

ThreeDWorld

A Platform for Interactive Multi-Modal Physical Simulation

• high-fidelity sensory data
• Physical interactions between mobile agents and objects in rich 3D environments.
• real-time near-photorealistic image rendering; generative procedures
• high-fidelity audio rendering;
• realistic physical interactions for a variety of material types, including cloths, liquid, and deformable objects
• customizable “agents” that embody AI agents;
• support for human interactions with VR devices. TDW’s API enables multiple

​

​

37

37

ThreeDWorld

38

38

ScenseScript

39

https://ai.meta.com/blog/scenescript-3d-scene-reconstruction-reality-labs-research/

39

Imitating Shortest Paths in Simulation

​

Effective Navigation and Manipulation in the Real World

​

40

40

LEARNING RIGID DYNAMICS WITH FACE INTERACTION GRAPH NETWORKS

​

• Simulating rigid collisions among arbitrary shapes is difficult
• complex geometry
• strong nonlinearity of the interactions
• (GNN)-based models learn to simulate complex physical dynamics
• fluids
• cloth
• articulated bodies
• “Face Interaction Graph Network” (FIGNet) extends beyond GNN-based methods,

​

​

41

41

Modeling faces not just nodes

42

42

Graph Networks as Learnable Physics Engines

43

43

Follow Kelsey Allen

44

https://k-r-allen.github.io/

44

Follow Alvaro Sanchez Gonzalez

45

45

Unisim: Probably the LLM equivalent for simulators

Learning Interactive Real-World Simulators

What is the difference with Sora?

• Unisim is unified action-in-video-out generative framework
• Combining diverse datasets rich in along different dimensions e.g., objects, scenes, actions, motions
• Formulate the action-in-video-out framework as an observation prediction model conditioned on finite history and parametrized by a video diffusion model
• Enable both high-level language policies, low-level control policies, and video captioning models to generalize to the real world when trained purely in simulation, thereby bridging the sim-to-real gap

46

46

Unisim

47

47

Other resources for Physics Simulation @NeurIPS 2023

48

48

Climate simulations with AI

• https://www.climatechange.ai/events/iclr2024
• https://www.climatechange.ai/events/neurips2023#schedule

49

49

A taxonomy of simulation research papers

50

Molecular/DNA/Proteins

Newtonian based

Rigid body

particles

Astronomical Scale

Human scale

Thermal physics

GNN based

Neural ODE/PDE

Generative based

Logic Rules

Videos Images

Quantum particles

Fluid Dynamics

Other simulations

50

Other NeurIPS 2023 resources relevant to physics

AI for Science: from Theory to Practice

1. Open Catalyst Challenge: Use AI to model and discover new catalysts to address the energy challenges posed by climate change
2. Predicting the Initial Conditions of the Universe using a Deterministic Neural Network
3. Towards LLMs as Operational Copilots for Fusion Reactors
4. Surrogate Modeling for Computationally Expensive Simulations of Supernovae in High-Resolution Galaxy Simulations
5. Augmenting large language models with chemistry tools
6. AdsGT: Graph Transformer for Predicting Global Minimum Adsorption Energy
7. Using the Transformer Model for Physical Simulation: An application on Transient Thermal Analysis for 3D Printing Process Simulation
8. Fast and Scalable Inference of Dynamical Systems via Integral Matching
9. Rapid Prediction of Two-dimensional Airflow in an Operating Room using Scientific Machine Learning
10. Latent Task-Specific Graph Network Simulators
11. Latent Neural PDE Solver for Time-dependent Systems
12. Self-supervised Learning to Discover Physical Objects and Predict Their Interactions from Raw Videos
13. Sensitivity Analysis of Simulation-Based Inference for Galaxy Clustering
14. Van der Pol-informed Neural Networks for Multi-step-ahead Forecasting of Extreme Climatic Events

51

51

Other NeurIPS 2023 resources relevant to physics

Machine Learning and the Physical Sciences

• Towards an Astronomical Foundation Model for Stars
• KeyCLD: Learning Constrained Lagrangian Dynamics in Keypoint Coordinates from Images
• When Black-box PDE Solvers Meet Deep Learning: End-to-End Mesh Optimization for Efficient Fluid Flow Prediction
• Simulation-based Inference for Cardiovascular Models
• Fast SoC thermal simulation with physics-aware U-Net
• Attention-guided neural differential equations for physics-constrained deep learning of ion transport
• Data-Driven Autoencoder Numerical Solver with Uncertainty Quantification for Fast Physical Simulations
• Incremental learning for physics-informed neural networks
• Modeling Coupled 1D PDEs of Cardiovascular Flow with Spatial Neural ODEs
• Redefining Super-Resolution:Fine-mesh PDE predictions without classical simulations
• Physical Symbolic Optimization
• Locating Hidden Exoplanets Using Machine Learning
• Optimized Dry Cooling for Solar Power Plants
• Trick or treat? Evaluating stability strategies in graph network-based simulators
• DeepSurveySim: Simulation Software and Benchmark Challenges for Astronomical Observation Scheduling
• Accelerating Flow Simulations using Online Dynamic Mode Decomposition
• Classification under Prior Probability Shift in Simulator-Based Inference: Application to Atmospheric Cosmic-Ray Showers
• Understanding and Visualizing Droplet Distributions in Simulations of Shallow Clouds
• Application of Zone Method based Physics-Informed Neural Networks in Reheating Furnaces

52

52

19 Main track + 11 Benchmark papers (simulation)

1. AVOIDDS: Aircraft Vision-based Intruder Detection Dataset and Simulator
2. Benchmark of Machine Learning Force Fields for Semiconductor Simulations: Datasets, Metrics, and Comparative Analysis
3. ScenarioNet: Open-Source Platform for Large-Scale Traffic Scenario Simulation and Modeling
4. SARAMIS: Simulation Assets for Robotic Assisted and Minimally Invasive Surgery
5. Neural Lighting Simulation for Urban Scenes
6. Equivariant Spatio-Temporal Attentive Graph Networks to Simulate Physical Dynamics
7. DeepSimHO: Stable Pose Estimation for Hand-Object Interaction via Physics Simulation
8. Calibrating Neural Simulation-Based Inference with Differentiable Coverage Probability
9. AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
10. Expressivity-Preserving GNN Simulation
11. MoCapAct: A Multi-Task Dataset for Simulated Humanoid Control

Before 2023

• LIPS - Learning Industrial Physical Simulation benchmark suite
• SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning
• NeurOLight: A Physics-Agnostic Neural Operator Enabling Parametric Photonic Device Simulation
• When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning
• DeepFoids: Adaptive Bio-Inspired Fish Simulation with Deep Reinforcement Learning
• A composable machine-learning approach for steady-state simulations on high-resolution grids
• Learning Modular Simulations for Homogeneous Systems
• Differentiable Simulation of Soft Multi-body Systems
• Learning to Simulate Self-driven Particles System with Coordinated Policy Optimization

53

53

Nick’s favorite Simulation framework

54

https://ai4abm.org/

54

Back to Reasoning with simulators

55

55

Probabilistic Programs as the language of simulation

56

56

Detailed info

57

[2306.12672] From Word Models to World Models

57

Language models as goal/reward

58

58

Guiding simulators with a structured language

59

Language to Rewards for Robotic Skill Synthesis

59

Limitations

60

60

Multi Modal Theory of Mind

61

MMToM-QA: Multimodal Theory of Mind Question Answering

61

Language Models for social reasoning

62

62

Games as simulators

63

Voyager: An Open-Ended Embodied Agent with Large Language Models

63

64

64

65

65

66

66

67

67

Social Simulator

68

Training Socially Aligned Language Models on Simulated Social Interactions

68

Imagining and Verbalizing

69

DreamLLM: Synergistic Multimodal Comprehension and Creation

69

70

70

An example

71

71

Takeaways

72

72

Distributed LLMs

Composing Big models from smaller ones

73

73

The LLM as a giant vector

• Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models
• Efficiently Tuned Parameters are Task Embeddings
• Editing Models with Task Arithmetic
• An Empirical Study of Multimodal Model Merging
• TIES-Merging: Resolving Interference When Merging Models
• Soft Merging of Experts with Adaptive Routing
• PETALS: Collaborative Inference and Fine-tuning of Large Models
• Git Re-Basin: Merging Models modulo Permutation Symmetries
• ColD Fusion: Collaborative Descent for Distributed Multitask Finetuning
• Fusing finetuned models for better pretraining
• Patching open-vocabulary models by interpolating weights
• Multitask Prompted Training Enables Zero-Shot Task Generalization
• Merging Models with Fisher-Weighted Averaging
• Merge Large Language Models with mergekit

74

74

The LLM as a giant vector

75

75

Would you ever write your product as a huge C++ file?

• The Software 2.0 paradigm refers to small models (a single cpp file)
• We need to be able to “compile” each model independently
• Finally build a big model by combining them
• We also need to be able to track changes

Let’s see what the software 3.0 might look like

76

76

Software 3.0

• Data curation (1.0: Writing the code, vscode)
• Small Language Model training (1.0: Compilation, gcc )
• Libraries of Language Models (1.0: Library making gcc -fpic)
• Linking Language Models for a specific task (1.0: Linking gnu ld)
• Tracking changes (1.0: git)

77

77

Building a bigger Language Model

78

78

Distribution of Heterogeneous LLM

79

79

Collaborating on LLM training (Git)

80

80

A taxonomy of LLM merging

81

81

Incremental maintenance of LLMs

82

82

Building an LLM from scratch

83

83

Democratizing LLM building

• Are the capabilities of LLMs bound by data? (see Beyond Scaling Panel)
• Attributing Model Behavior at Scale (ATTRIB)
• Contributing to an Efficient and Democratized Large Model Era [slides]
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model

​

84

84

All the necessary plumbing for building LLMs

Infrastructure is not easy

85

85

Hardware cannot keep up with model growth

86

86

Startups will be able to afford building GPT-3 in 2 years

87

87

Colossal-AI contribution

88

88

It takes a village to build one of them

89

burns

89

Colossal-AI offers a framework for building a LLM

90

90

Looks like there is a lot of demand!

91

91

Congratulations!

You build your first really Large LM

Can you tune, maintain, grow it without redoing everything from scratch?

​

92

92

Attribution as a dimension for optimizing LLM costs

93

93

Pillar I: Data attribution

94

94

Pillar II: Model Attribution

95

95

Pillar III: Algorithm Attribution

96

96

DataInf

97

DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models

97

Tiny LMs might be the solution

98

• TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
• Faster training/tuning cycles
• Easier to do ablation studies on architecture and data contribution
• You can tune an XGB model because it takes an hour to train it
• If you can train a tiny LLM in hours then you can tune it too
• Like XGB feature engineering the art is in crafting a good dataset

​

98

Mathematics/Reasoning and LLMs

99

99

The most exciting topic

• How far are we from deepmath?
• Machine Reading of the math web with LLMs
• Can we follow the same paradigm for common and other reasoning?
• Revisiting the Lean prover

​

100

100

Mathematical Reasoning

• MATH-AI: The 3rd Workshop on Mathematical Reasoning and AI [site]
• Machine Learning for Theorem Proving [slides]
• Advancing mathematics by guiding human intuition with AI
• Llemma: An Open Language Model For Mathematics
• CLOVER: CLOSED-LOOP VERIFIABLE CODE GENERATION
• LeanDojo: Theorem Proving with Retrieval-Augmented Language Models
• Draft, sketch, and prove: Guiding formal theorem provers with informal proofs
• STaR: Bootstrapping Reasoning With Reasoning
• Thinking Like Transformers
• What Algorithms can Transformers Learn? A Study in Length Generalization
• Tractable Control for Autoregressive Language Generation
• On the Paradox of Learning to Reason from Data
• IMAGE INPAINTING VIA TRACTABLE STEERING OF DIFFUSION MODELS
• Large Language Models as Analogical Reasoners
• Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models

​

​

101

101

Deepmind’s recent buzz

102

102

What is formal theory proving?

103

103

Code vs. math symbols

104

104

LLMs are doing ok on high school level

105

105

LLema vs. Minerva

106

106

Informal vs. Formal Mathematical Reasoning

107

107

Checking Mathematical Proofs is Hard for Humans

108

108

Proof Assistants (Interactive Theorem Provers)

109

109

Examples of Proof Assistants

110

110

Generating Proof Steps (Tactics)

111

111

Searching for Proofs

112

112

Best First Search

113

113

Hyper Tree Proof Search

114

114

Is Proof Search Really Necessary?

115

115

Premise Selection

116

116

Magnushammer

117

117

Reprover: Retrieval-Augmented Prover

118

118

Premise Retrieval Improves Theorem Proving

119

119

LeanDojo

120

120

From Informal to Formal Proofs

121

https://www.newscientist.com/article/2422601-mathematicians-plan-computer-proof-of-fermats-last-theorem/

121

The ecosystem

122

122

The process

123

123

An example

124

124

Theorem provers for code verification

With the help of LLMs

125

125

How can we verify code produced by LLMs?

126

126

Theorem Proving for Verified Code Generation

127

127

Formal Software Verification

128

128

Software verification on the wild

129

129

Current tools

130

130

Proof Synthesis SoTA

131

131

Proofster

132

132

The maintenance problem

133

133

How about an LLM?

134

134

The Clover Paradigm

135

135

Theorem Proving and LLMs: Takeaways

136

136

MATH-AI: The 3rd Workshop on Mathematical Reasoning and AI

137

137

138

138

Generating Reasoning data with LLMs for finetuning LLMs

139

139

Analogical prompting

140

140

A math example (1)

141

141

A math example (2)

142

142

Benchmarks

143

143

More Benchmarks

144

144

Discussion: abstraction is key in analogical reasoning

145

145

Takeaways

146

146

Literature

147

147

The trend continues

148

SELF-DISCOVER: Large Language Models Self-Compose Reasoning Structures

148

A programming Language for Transformers RASP-L

149

149

The original Language RASP

150

https://github.com/yashbonde/rasp

150

RASP-L extension

151

151

The paradox of Learning to reason from data

152

152

What can BERT learn?

153

153

BERT prefers Statistical to Logical Thinking

154

154

How should we interpret this phenomenon?

155

• Are we using the wrong paradigm for training our models?
• Is it impossible for logic to emerge from statistics?
• What is the architecture that creates logic from statistical observations? (apart from the human brain)
• Is that the definition of AGI?

155

Injecting Logic to GenAI models

156

156

More examples

157

157

Literature

158

158

Injecting Logic in the transformer

159

159

Symbol processing required for implication rule

160

160

Symbol processing required for implication rule

161

161

Symbol processing required for implication rule

162

162

Transformer Production Framework

163

163

Symbol processing required for implication rule

164

164

What we have learned

165

165

Biases for a new generation of deep-reasoning LMs

166

166

LLMs for Tabular Data

167

167

Table representation Learning Workshop

• TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second
• IngesTables: Scalable and Efficient Training of LLM-Enabled Tabular Foundation Models
• How to Prompt LLMs for Text-to-SQL: A Study in Zero-shot, Single-domain, and Cross-domain Settings
• Data Ambiguity Strikes Back: How Documentation Improves GPT's Text-to-SQL
• MultiTabQA: Generating Tabular Answers for Multi-Table Question Answering
• Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering
• Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection
• MotherNet: A Foundational Hypernetwork for Tabular Classification

​

​

Invited talk: Advances in In-Context Learning for Tabular Datasets

Invited talk: Next-Generation Data Management with Large Language Models

168

168

Tables are everywhere

169

169

Goals of the TRL workshop

170

170

Last year’s best paper

171

171

Very Interesting Results

172

172

TabPFN 1 year ago

173

173

TabPFN 2.0

174

174

GPT-4 as Data Science Assistant

175

175

An example

176

176

Benchmarks

177

177

Feature Engineering

178

178

Takeaways from CAAFE

179

179

Can LLMs learn how to do Gradient Descent?

180

180

In-Context Learning

181

181

A surprising Experiment

182

182

Literature

183

183

Prior-Fitted Networks (PFNs) Visualized

184

184

Quantitative result (87 numerical datasets without missing values)

185

185

Performance with many objectives

186

186

MotherNet

187

187

Generate a model instead of predictions

188

188

Architecture

189

189

Literature

190

190

Conclusions

191

191

Takeaways on TabPFN

192

192

The LLM version of Kumo

193

193

Another approach to the same problem

194

194

Text to SQL

What we have learned so far

195

195

Lessons learned from Natural Language to SQL

196

196

How Documentation improves GPT’s Text-to-SQL

197

197

Another Text-To-SQL paper

198

198

Benchmarks

199

199

New frontiers in Graph Learning

200

200

Foundational Models for graphs and relational data

201

201

The new Web

202

202

RAG to the rescue

203

203

Application Development with LLMs

204

204

Elementary and advanced

• Application Development using Large Language Models
• Instruction Tuning and Instruction Following [site]
• 4th Workshop on Self-Supervised Learning: Theory and Practice [site]

205

205

Wrapping up

206

206

Where do we go next?

• We celebrated the 10 year birthday of LLMs
• We celebrated the one year birthday of ChatGPT
• Yet OpenAI was not present at NeurIPS (Why?)
• Research community clearly sides with open source LLMs
• The two main challenges
• More good data
• Cheaper Models
• Cheaper hardware?
• Less Parameters?
• More Efficient algorithms?
• Will AI push math to its limits?

207

207

208

208