AI Safety Fieldbuilding

ALIGN Japan – Ryan Kidd

Ryan Kidd

2017-2022 PhD in Physics, UQ

2022-now Co-Director, MATS

2023-now Co-Founder, LISA

2023-now Regrantor, Manifund

​

Artificial general intelligence (AGI)

1. What is it?
2. When will it be here?
3. How will it change the world?

Artificial general intelligence (AGI)

• What is it?
• When will it be here?
• How will it change the world?

Metaculus criteria:

• Passes 2 h adversarial Turing test
• Assembles model car
• 75% on all MMLU tasks + 90% on avg.
• 90% top-1 accuracy on APPS

Artificial general intelligence (AGI)

• What is it?
• When will it be here?
• How will it change the world?

Artificial general intelligence (AGI)

• What is it?
• When will it be here?
• How will it change the world?

Artificial general intelligence (AGI)

• What is it?
• When will it be here?
• How will it change the world?

Automation of labor

Artificial general intelligence (AGI)

• What is it?
• When will it be here?
• How will it change the world?

Economic growth

Transformative AI: causes 20-30% GDP growth per year

Doubling time: ~3 years

Artificial general intelligence (AGI)

• What is it?
• When will it be here?
• How will it change the world?

Societal change

Types of superintelligence:

Quality: Smarter than humans

Speed: Faster than humans

Collective: More numerous or organized than humans

Mission

• AGI may be dangerous and near
• AI safety is talent-constrained
• We can help!

Metaculus: 9% AI x-risk by 2100

Mission

• AGI may be dangerous and near
• AI safety is talent-constrained
• We can help!

Mission

• AGI may be dangerous and near
• AI safety is talent-constrained
• We can help!

Manhattan

Apollo

AGI

TAI

+28% per year

Mission

• AGI may be dangerous and near
• AI safety is talent-constrained
• We can help!

Three futures

1. Privatized AGI
2. Nationalized AGI
3. AGI winter

Three futures

• Privatized AGI
• Nationalized AGI
• AGI winter

AI Safety Institutes

Three futures

• Privatized AGI
• Nationalized AGI
• AGI winter

Three futures

• Privatized AGI
• Nationalized AGI
• AGI winter

Three futures

Privatized AGI AI gov + evals + infosec + lab safety teams

Nationalized AGI International coalition building + open source AI alignment

AGI winter “Provably safe AI” + all the above

What can we do?

ALIGN Japan

Goals

• Accelerate high-impact scholars
• Support high-impact mentors
• Grow AI safety research field

What do new researchers need?

• Strong technical skills
• High-quality mentorship
• Basic understanding of threat models
• Community + support
• For jobs: publications, fast coding

What do mentors need?

• Research assistants
• Hiring pipeline
• Experience as a manager
• Support to scale projects

What does the AI safety field need?

• Connectors: Build new paradigms
• Iterators: Further current paradigms
• Amplifiers: Scale projects

What does the AI safety field need?

• Connectors: Paul Christiano, Buck Shlegeris, Evan Hubinger, Alex Turner, etc.
• Iterators: Ethan Perez, Neel Nanda, Dan Hendrycks, etc.
• Amplifiers: Bill Zito, Oliver Zhang, Emma Abele, Chris Akin, etc.

What does the AI safety field need?

MATS’ strategy

ALIGN Japan

Program elements

MATS Program

• Mentored research projects
• Berkeley office + community
• Seminars + workshops
• Research management
• Extension program (up to 4 months)

Evaluation milestones

• Jul 19: Research plan (2-pages)
• Theory of change
• Timeline + output

​

• Aug 23: Symposium talk (5 min + Q&A)
• What did you do and why?
• Results + relevance

​

• Aug 23: Mentor reports + scholar self-reports

Research management

• Help scholars set goals, be productive, and use mentor time well
• Help mentors track scholar research
• Help MATS achieve program outcomes

How have we done so far?

ALIGN Japan

MATS history

• 213 scholars
• 47 mentors

What are we doing now?

MATS Team Orientation

MATS 6.0 research interest

• 1221 scholar applications
• 60 mentor applications

MATS 6.0 research portfolio

and more!

MATS 6.0 mentors

Interpretability

• Neel Nanda & �Arthur Conmy
• Lee Sharkey
• Alex Turner
• Stephen Casper
• Adrià Garriga Alonso
• Jesse Hoogland
• Lawrence Chan

​

• Erik Jenner
• Jason Gross
• Jessica Rumbelow
• Adam Shai
• Nandi Schoots
• Lucius Bushnaq & �Jake Mendel

Oversight + control

• Ethan Perez & Akbir Khan
• Fabien Roger
• Buck Shlegeris
• David Lindner
• Sebastian Farquhar
• Mantas Mazeika
• Scott Emmons
• Shi Feng

MATS 6.0 mentors

Evaluations

• Evan Hubinger
• Owain Evans
• Hjalmar Wijk
• Marius Hobbhahn
• Jérémy Scheurer
• Francis Rhys Ward
• Steven Basart
• Nico Miailhe

Governance

• Timothy Fist
• Mauricio Baker
• Lisa Thiergart
• Matija Franklin & Philip Moreira Tomei

Value alignment

• Micah Carroll
• Brad Knox
• Tsvi Benson-Tilsen

​

Cooperative AI

• Christian Schroeder de Witt
• Jan Kulveit

​

Provably safe AI

• Yoshua Bengio & MILA

MATS 6.0 scholars

Median: Male, 26, Masters student,� 520/600 on CodeSignal

18%

81%

Ryan’s Manifund requests for proposals

• AI safety PhDs (e.g., with these supervisors)
• AI safety PhD advisory service
• Research on AI safety macrostrategy
• AI safety hubs outside of the SF Bay Area or London
• AI safety start-up incubators

AI Safety Fieldbuilding

ALIGN Japan – Ryan Kidd