Irina Rish

Mila - Quebec AI Institute

University of Montreal

Scaling to AGI�Scaling, Emergence, Open Questions

​

Canada Excellence Research Chair

in Autonomous AI

CERC-AAI Lab: irina-lab.ai

​

• Out-of-Distribution (OoD) Generalization
• Transfer learning
• Domain Adaptation
• Meta-learning
• Continual learning
• Adversarial Robustness

Generalization: The “Holy Grail” of ML

AGI ⇔ “General” AI ⇔ Multi-task,“Broad” AI

“Highly autonomous systems that outperform humans at most economically valuable work” (OpenAI definition)

Emergent Compositional Generalization?

Learning World’s “Principal Components”

f1(x) f2(x) f3(x) … fn(x) …

Assumption: all data & future tasks are “well-approximable” in some finite “basis”

{ h1(x) , … , hk(x) }

Foundation Models & Scaling Revolution (2020-?)

“Train one model on a huge amount of data and adapt it to many applications.

We call such a model a foundation model.”

​

CEFM: Stanford’s Center for Research on Foundation Models

“On the Opportunities and Risks of Foundation Models”

Application example: healthcare

AI & Scaling

“The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin."

The Bitter Lesson (Rich Sutton, March 13, 2019)

Scaling Laws as “Investment Tools” for AI

An example:

image transformers dominated by convnets in lower data regimes, but outperforming the latter with more data: https://arxiv.org/pdf/2010.11929.pdf

1. Universal scaling laws govern the behavior and structure of complex systems, from biological organisms to cities and companies

​

• Living organisms, companies, cities all exhibit predictable behaviors based on size but have different dynamics due to their unique internal structures and the role of innovation.

​

• Living organisms follow specific scaling laws: e.g. metabolic rates, lifespans, and growth patterns scale predictably with size.

​

• Cities scale differently: their productivity and innovation increase disproportionately, but so do problems like crime and pollution. This is in contrast to biological systems, where larger size leads to slower metabolic rates.

​

• Companies initially grow like biological systems, but they don’t scale indefinitely. Unlike cities, they tend to plateau and eventually decline. Innovation is crucial for sustaining long-term growth in organizations.

​

Predicting Network’s Behavior at Scale

​

• “Closed-box” predictions (empirical scaling laws)
• “Open-box” predictions (observing learning dynamics)

Neural Scaling Laws: Kaplan et al

Jared Kaplan et al, Scaling Laws for Neural Language Models, 2020.

Scale and Inductive Biases

Brief History of Neural Scaling Laws

Kaplan et al, Scaling Laws for Neural Language Models, 2020

Cortes et al. Learning curves: Asymptotic values and rate of convergence. NeurIPS 1994.

First to observe power law scaling: of ANNs:

​

x = dataset size and y = test error.

Hestness et al. Deep Learning Scaling is Predictable,Empirically. Dec 2017.

​

​

​

​

1994

2017

Showed that data-size dependent scaling laws given by power laws hold over many orders of magnitude.

Rosenfeld et al. . A constructive prediction of the generalization error across scales. 2019.

Applied power laws to model-size dependent scaling laws, i.e. when x = number of parameters.

​

Showed that power law applies when x = compute, besides x = data and x = model.

This paper brought “neural” scaling laws to the mainstream as it was in context of GPT-3 training.

2019

2020

Chinchilla Scaling Laws

”Chinchilla's wild implications”, AI Alignment Forum, July 30th, 2022

“Data, not size, is the currently active constraint on language modeling performance”

“Old” GPT Scaling Laws

(NOT compute-optimal!):

Data/Model = 2/5

Chinchilla Scaling Laws: Data/Model = 50/50

Chinchilla Scaling Laws

Same amount of compute:

Chinchilla (70B parameters, 1.4T tokens)

Gopher (280B parameters, 300B tokens)

Beyond Chinchilla:

Focus to Inference Costs

• Post-Chinchilla Scaling: Llama etc

​

• Inference scaling laws (2024)

​

• Quantization, compression, low-bit model training (Spectra etc)

​

LLaMA-13B outperforms GPT-3 (175B) on most benchmarks

​

LLaMA-65B is competitive w/ best models: Chinchilla-70B, PaLM-540B

​

Beyond Chinchilla compute-optimal, towards “inference-optimal”

LLaMA: Open and Efficient Foundation Language Models

Llama 2: Open Foundation and Fine-Tuned Chat Models

More Data Needed - No Saturation in Sight!

Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scale

(Kaushal et al, 2024)

Reducing memory and inference costs by training low-bitwidth models

​

Spectra: “Small but Mighty” Quantized LLMs

(ICLR 2025 Spotlight, ACL)

Beyond Chinchilla-optimal: Accounting for inference in language model scaling laws (Sardana et al, ICML 2024)

Beyond Chinchilla-optimal: Accounting for inference in language model scaling laws (Sardana et al, ICML 2024)

Chinchilla parametric scaling curves fit to “typical” (smaller) token/parameter ratios overestimate the impact of additional data.

​

None of of our parametric curves fit 150M long-ratio training results well; as we extend training duration far beyond typical Chinchilla ratios, the parametric loss function is not flexible enough.

​

Alternative functional forms?�BNSL?

More Complex Scaling Behavior:

“Phase Transitions”, Emergent Phenomena

• 3SAT, CSPs, NPhard problems
• Random graphs
• Universal Laws of Robustness
• GPT-3 on Arithmetic task
• Grokking

-

Predicting Network’s Behavior at Scale

​

• “Open-box” predictions (observing learning dynamics)
• “Closed-box” predictions (empirical scaling laws)

Transition from Memorization to Generalization

Notsavo et al, Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grok. 2023

Initial stage/”confusion” (t0 to t1)

Memorization(t1-t2)

Comprehension (t3-t4)

Generalization (“grokking”) at t4+

​

t2

t3

t1

t4

Spectral Signature of Loss Predicts Grokking

Notsavo et al, Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grok. 2023

no grokking

(after 10k steps)

grokking

Predicting Network’s Behavior at Scale

​

• “Open-box” predictions (observing learning dynamics)
• “Closed-box” predictions (empirical scaling laws)

Broken Neural Scaling Laws:

A Universal Functional Form for Neural Scaling Laws?

​

Ethan Caballero et al, 2022

https://arxiv.org/abs/2210.14891

Sparse Models

Distillation

Diffusion Models

Alignment (ELO score)

Reinforcement Learning

Coding

Video

BNSL accurately fits and extrapolates a very wide range of scaling behaviors

• Settings: Zero-Shot, Prompted, and Fine-Tuned settings; Downstream and upstream
• Tasks: Large-Scale Vision, Language, Audio, Video, Diffusion, Generative Modeling, Multimodal Learning, Contrastive Learning, AI Alignment, AI Capabilities, Robotics, Out-Of-Distribution Generalization, Continual Learning, Transfer Learning, Uncertainty Estimation / Calibration, Out-Of-Distribution Detection, Adversarial Robustness, Distillation, Sparsity, Retrieval, Quantization, Pruning, Fairness, Molecules, Computer Programming/Coding, Math Word Problems, Arithmetic, Double Descent, “Emergent” “Phase Transitions”, Supervised Learning, Unsupervised / Self-Supervised Learning, & Reinforcement Learning (Single Agent & Multi-Agent)
• Architectures: ResNet, Transformer, MLP-Mixer, MLP, Graph Neural Network, U-Net, Ensemble, Sparsely-Gated Mixture-of-Experts, Sparse Pruned Model
• X-axes: Compute, Dataset Size, Number of Model Parameters, Number of Training Steps, Input (e.g. Context) Size, & Upstream Performance
• Y-axes: prediction error, cross entropy, calibration error, AUROC, BLEU score percentage, F1 score, reward, Elo rating, FID score

BNSL accurately extrapolates the scaling behavior of:

Non-Monotonic Scaling (e.g. Double Descent)

Inflection Points

(e.g. Four Digit Addition)

Control:

Model: size N, architecture,...

Data: size D, diversity,...

No control:

Downstream task(s) complexity (T)

Relative Scaling Laws?

Scaling laws:

not just L(N,D), but rather L(N,...,D,...| T) ?

Training Foundation Models

“We think the most benefits will go to whoever has the biggest computer.” � Greg Brockman, OpenAI’s CTO, Financial Times

Most compute is owned by AI companies (Google, OpenAI, etc), not academia & nonprofit research; this “compute gap” continues to widen.

​

​

Open Foundation Models on Supercomputers

News

5.9M V100 GPU hrs on Summit

Ongoing CERC-AAI Lab Projects

Language Models: Pretraining and Continual Learning

Aligned Multimodal Language-Vision Models:

Time-series Transformers

Compression/Distillation & Fast Inference

Generalist Agents: Open Gato & Open Ada

LLM 4 Psychology & Psychology 4 LLMs

​

Kshitij Gupta

Daniel Kaplan

Quentin Anthony

Arjun Ashok

Benjamin Thérien

Tejas Vaidhya

Adam Ibrahim

Andrew WIlliams

Alexis Roger

Costs of Scaling AI Models

Source: epochai.org

The total amortized cost of developing Gemini Ultra, including hardware, electricity, and staff, is estimated at $130 million

Training compute of frontier AI models grows by 4-5x per year

What’s Next?

Alternatives to Transformers?

​

​

Better Scaling / More Efficient Models?

AI & Scaling

“We want AI agents that can discover like we can, not which contain what we have discovered. Building in our discoveries only makes it harder to see how the discovering process can be done.”

​

The Bitter Lesson (Rich Sutton, March 13, 2019)

Scaling Biological Networks -> Scaling AI?

​

• Evolution scaled biological networks from single cells to brains

​

• What are the “scaling algorithms” and “scaling laws”?

​

• Emergence and transitions in bio networks:

both at developmental and evolutionary scale

​

• what can AI researchers working on scaling AI while ensuring its safe and steerable learn from nature about emergence of novel behaviors?

​

​

Thank you!

Expanding Scaling Laws

• Synthetic data scaling laws? �Scaling laws of synthetic images for model training... for now

Synthetic images demonstrate a scaling trend similar to, but slightly less effective than, real images in CLIP training, while they significantly underperform in scaling when training supervised image classifiers