1 of 41

Advice for a (young) investigator in the first and last days of the Anthropocene

Jascha Sohl-Dickstein�Anthropic

[h/t Tim Urban, Wait But Why]

Title: Advice for a young investigator in the first and last days of the Anthropocene

Abstract: Within just a few years, it is likely that we will create AI systems that outperform the best humans on all intellectual tasks. This will have implications for your research and career! I will give practical advice, and concrete criteria to consider, when choosing research projects, and making professional decisions, in these last few years before AGI.

--

The title is a play on a very opinionated and pragmatic book by another nobel prize winner, ramon y cajal, who is also one of the founders of modern neuroscience.

To get you in the right mindset, on the right we have a plot of GDP vs time.

That is you, standing precariously on the top of that curve.

You are thinking to yourself -- I live in a pretty normal world.

Some things are going to change, but the future is going to look mostly like a linear extrapolation of the present.

And the plot should suggest that this may not be the right perspective on the future.

This plot by the way looks surprisingly similar even if you plot it on a log scale. We didn't stabilize on our current rate of growth until around 1950.

Even if the exponential held all the way back, not just back to 1950 -- you should still predict a weird future. Like, colloquially, if someone said "we're experiencing exponentially increasing change", you should expect that to mean there is going to be large magnitude change.

colab link: https://colab.research.google.com/drive/1FLov1kDDa_NF7m_XxjMPaJ3IS_HePolQ?usp=sharing

2 of 41

AI is transforming the world. Now.
You should take that seriously in your career and research.
You have immense leverage on how the whole thing turns out.��
... and a few practical suggestions

3 of 41

Anthropocene

Proposed geologic epoch characterized by impact of human activities on the Earth (cf. 6th mass extinction)

Started around 1950

Let's talk about geologic epochs.

They are a measure of deep time, divided by distinct changes in life forms, climate, and geological processes, that can be observed in rock strata.

Humans are starting to have geologic impact. We are causing the 6th mass extinction in the history of the earth. Radioisotopes from nuclear weapons tests are being captured in rock strata. You will be able to tell, a billion years in the future, that we were here.

There's a proposal to name this current period of human driven global change the Anthropocene epoch.

This focus on human-driven change is interesting, because we are perhaps near the end of the period of humans being the primary intellects driving global change.

The central conceit of the talk is that, very much depending on the way in which AI plays out, this could be a very, very short geologic epoch.

source: https://www.futurelearn.com/info/courses/taking-on-the-climate-crisis/0/steps/261255

4 of 41

Anthropocene

Proposed geologic epoch characterized by impact of human activities on the Earth (cf. 6th mass extinction)

Started around 1950

?

Let's talk about geologic epochs.

They are a measure of deep time, divided by distinct changes in life forms, climate, and geological processes, that can be observed in rock strata.

Humans are starting to have geologic impact. We are causing the 6th mass extinction in the history of the earth. Radioisotopes from nuclear weapons tests are being captured in rock strata. You will be able to tell, a billion years in the future, that we were here.

There's a proposal to name this current period of human driven global change the Anthropocene epoch.

This focus on human-driven change is interesting, because we are perhaps near the end of the period of humans being the primary intellects driving global change.

The central conceit of the talk is that, very much depending on the way in which AI plays out, this could be a very, very short geologic epoch.

source: https://www.futurelearn.com/info/courses/taking-on-the-climate-crisis/0/steps/261255

5 of 41

AI progress in the Anthropocene

approximate human brain lifetime compute

~ start of Anthropocene

Here is a plot of the amount of compute used to train notable AI models, starting at the beginning of the Anthropocene, through the present. You can see a change in slope around the point where AI models became economically valuable.

Each tick on the y axis is two orders of magnitude larger than the one below it.

In the upper right I've circled a rough plausible range for the compute performed by the human brain in a lifetime.

My own back of the envelope estimate, assuming one floating point operation per synapse per millisecond, is 1 trillion petaflops, or a fraction of a grid division larger than current models.

So we are reaching scales of model training compute similar to those in the human brain across a lifetime.

Though not necessary or sufficient, approaching human levels of compute makes it more plausible that we are approaching human levels of intelligence

6 of 41

AI progress in the Anthropocene

It's not just the scale of AI model compute that has been increasing.

Here's a recent study from metr looking at the timescale of diverse software tasks which AI models can perform unaided, with 50% success. The timescale is derived by looking at how long it takes humans to perform the same tasks.

here on the x-axis we have the year the model was released. Here on the y-axis we have the task length that the model can succeed at with 50% probability.

The key thing to observe is that it is increasing exponentially. If we extrapolate, we would predict that models can perform a full day's work (at least in software), unassisted, with 50% success, sometime in 2027.

The curves for higher success rate have the same slope, just a different intercept. So if you want 80% success, you should add a year or so to your AGI timeline.

At least in software

blog: https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/

paper: https://arxiv.org/abs/2503.14499

7 of 41

AI progress in the Anthropocene

8 of 41

AI progress in the Anthropocene

9 of 41

AI progress in the Anthropocene

Benchmark performance not explained by overfitting
Relative performance on new questions, mimicking those in GSM8K benchmark

[Zhang et al., 2024]

The first time I gave this talk, Yann Lecun was in the audience, and insisted that improvements on benchmarks were driven by benchmark questions being added to the training set.

Here is an experiment directly addressing this. A group of researchers at Scale created de novo questions, carefully matching the core skills and difficulties probed in an existing benchmark of math word problems.

Plotted is the relative performance of LLMs on the new questions, compared to on the original benchmark.

A y-axis of 0 would mean they did just as well on the new questions as the original benchmark questions. A y-axis of less than 0 means that the model did better on the original benchmark. That is, a y-axis of less than 0 suggests that the model was cheating, and was training on the test set. From this, you can see that there are labs that are training on the test set!

If you look closely though, all the labs that are cheating are the ones that are behind and struggling to catch up.

The frontier labs of Anthropic and OpenAI and mostly Google all do roughly as well on the new questions as they did on the old questions.

I'm particularly proud that Claude actually did even better on the new questions than the original benchmark. We are not overfitting to this benchmark.

grade school arithmetic reduc:

https://arxiv.org/html/2405.00332v1

10 of 41

AI progress in the Anthropocene

Benchmark performance not explained by overfitting

[AI] solutions were astonishing in many respects. IMO graders found them to be clear, precise and most of them easy to follow. -IMO President Prof. Dr. Gregor Dolinar

In case you think we are overfitting to specific benchmarks -- I should emphasize that it is not just academic benchmarks on which we see surprising breakthroughs.

Earlier this year, models from both Google and OpenAI achieved Gold Medal performance on the International Math Olympiad.

The IMO is a high school age math competition, that involves novel problems every year ... and it is really very, very hard. If you are not familiar with it, and don't believe it's hard, I suggest you try to do some of the problems, or ask a friend in the hard sciences about it. This year, in the whole world, 72 students achieved gold medals ... and two LLMs did.

Because these problems are made just for the competition, if an LLM gets them correct, it is not due to overfitting. There is no memorizing the answers.

Here's a quote from the competition organizer: "[AI] solutions were astonishing in many respects. IMO graders found them to be clear, precise and most of them easy to follow."

From conversations with people at multiple other labs, I think we're likely to see AI solving real open research problems in math in the next months. This isn't something I can show you yet -- but I would be happy to make bets on whether there will be published examples of models solving math problems no human has yet solved within the year.

And it's not just overfitting to academic benchmarks.

AI Performance on de novo tasks

11 of 41

AI progress in the Anthropocene

From: ⬛⬛⬛⬛⬛⬛ <⬛⬛⬛⬛⬛⬛⬛⬛⬛>

Date: Mon, 4 Aug 2025

Subject: Fwd: Aug 12 Special AM Seminar: "Advice for a young investigator in the first and last days of the Anthropocene"

To: jascha.sohldickstein@gmail.com

What a bat-shit crazy abstract...

The Overton window is moving

When I gave this talk at Harvard, I got this fascinating unsolicited email, where the entire message was "What a bat-shit crazy abstract...". (* the sender is now a postdoc at Princeton)

There's this concept called the Overton window. The Overton window is the set of ideas that are considered acceptable to express or publicly hold.

At least on the surface, this email is a statement that AGI is outside the overton window. Or at least that it's inside the Overton window to call predictions of AGI crazy.

Since the Overton window determines what we feel comfortable discussing, and even what we feel comfortable thinking, it's important to communicate that AGI is inside the overton window.

A primary purpose of this talk is to give you permission to take this stuff seriously.

What I am saying in this talk is normal. Respected and well known people and institutions say the same things, and they're still respected.

12 of 41

AI progress in the Anthropocene

The competition for AGI—AI that surpasses humans at all cognitive tasks—is of fundamental geopolitical importance. -Rishi Sunak, UK Prime Minister, 2025

As profound as [cell phone] technology has been, AI will be more impactful. And it is going to come faster ... We’re now starting to see these models, these platforms be able to perform really high level — what we consider to be really high level intellectual work. -Barack Obama, US president, 2025

We believe that in the next year, the vast majority of programmers will be replaced by AI. Within three to five years, we’ll see AGI systems as smart as the best humans. And in six years, artificial superintelligence smarter than all of us combined. This is happening fast, and society isn’t ready. -Eric Schmidt, former Google CEO, 2025

The Overton window is moving

13 of 41

AI progress in the Anthropocene

The competition for AGI—AI that surpasses humans at all cognitive tasks—is of fundamental geopolitical importance. -Rishi Sunak, UK Prime Minister, 2025

As profound as [cell phone] technology has been, AI will be more impactful. And it is going to come faster ... We’re now starting to see these models, these platforms be able to perform really high level — what we consider to be really high level intellectual work. -Barack Obama, US president, 2025

We believe that in the next year, the vast majority of programmers will be replaced by AI. Within three to five years, we’ll see AGI systems as smart as the best humans. And in six years, artificial superintelligence smarter than all of us combined. This is happening fast, and society isn’t ready. -Eric Schmidt, former Google CEO, 2025

What I am saying in this talk is normal. Respected and well known people and institutions say the same things, and they're still respected.

To illustrate this, let me read some quotes from respected people predicting disruption from AGI

"The competition for AGI—AI that surpasses humans at all cognitive tasks—is of fundamental geopolitical importance."

-Rishi Sunak, recent UK Prime Minister, 2025

https://www.economist.com/by-invitation/2025/07/16/you-dont-have-to-be-america-or-china-to-win-in-ai-says-rishi-sunak

As profound as that technology has been, AI will be more impactful. And it is going to come faster, and the reason is that its applications aren’t just in the communications and information area. It has potential application across the board. To some degree, it’s an extension of this long trend towards automation, but it’s not now just automating manufacturing processes or the use of robot arms. We’re now starting to see these models, these platforms be able to perform really high level — what we consider to be really high level intellectual work.

->

"As profound as [cell phone] technology has been, AI will be more impactful. And it is going to come faster ... We’re now starting to see these models, these platforms be able to perform really high level — what we consider to be really high level intellectual work." -Barack Obama, 2025

14 of 41

AI progress in the Anthropocene

The competition for AGI—AI that surpasses humans at all cognitive tasks—is of fundamental geopolitical importance. -Rishi Sunak, UK Prime Minister, 2025

As profound as [cell phone] technology has been, AI will be more impactful. And it is going to come faster ... We’re now starting to see these models, these platforms be able to perform really high level — what we consider to be really high level intellectual work. -Barack Obama, US president, 2025

We believe that in the next year, the vast majority of programmers will be replaced by AI. Within three to five years, we’ll see AGI systems as smart as the best humans. And in six years, artificial superintelligence smarter than all of us combined. This is happening fast, and society isn’t ready. -Eric Schmidt, former Google CEO, 2025

And it's not just individuals. Journalists are flirting with the idea of AGI too. There are stories and opinion page pieces in serious newspapers taking the idea seriously.

Here's the economist considering the economics of AGI

and here is the nytimes with a quite strongly worded title for an interview with Ben Buchanan, the former special adviser for artificial intelligence in the Biden White House.

If these people take the idea seriously, you have permission to take it seriously too!

"The competition for AGI—AI that surpasses humans at all cognitive tasks—is of fundamental geopolitical importance."

-Rishi Sunak, recent UK Prime Minister, 2025

https://www.economist.com/by-invitation/2025/07/16/you-dont-have-to-be-america-or-china-to-win-in-ai-says-rishi-sunak

As profound as that technology has been, AI will be more impactful. And it is going to come faster, and the reason is that its applications aren’t just in the communications and information area. It has potential application across the board. To some degree, it’s an extension of this long trend towards automation, but it’s not now just automating manufacturing processes or the use of robot arms. We’re now starting to see these models, these platforms be able to perform really high level — what we consider to be really high level intellectual work.

->

"As profound as [cell phone] technology has been, AI will be more impactful. And it is going to come faster ... We’re now starting to see these models, these platforms be able to perform really high level — what we consider to be really high level intellectual work." -Barack Obama, 2025

15 of 41

AGI when?

Consensus timeline in San Francisco AI labs

2-5 years
"long" timelines, 10-30 years

Most recent surveys of AI experts

Personal

Current Claude feels like inept but �knowledgeable + eager grad student
That wasn't true a year ago
It likely won't be true in another year

13 year drop 2022 → 2023 predictions

[Grace et al., 2024]

So models are getting better very fast. And I've set an ominous or promising tone by talking about the start and end of a geologic epoch. I've talked about how the idea of AGI is normal enough now, that you're allowed to believe in it in polite company. But when can we expect it to happen?

When can we expect models to be human or superhuman on different things? Especially when can we expect them to be able to automate the kinds of tasks that make up the majority of our work?

If you live in the SF AI bubble I live in, the consensus answer is just a few years. People describe themselves as having long timelines if they think it will take a decade.

This is unusual, because practitioners are the most aware of the practical barriers that limit progress in the short run. My partner is in biotech, and if you ask researchers in biotech about exponentially declining cancer rates, the first thing they do is give a long list of barriers to eradicating cancer. (and cancer rates are exponentially dropping -- it's amazing! I have an optional slide at the end on this)

So when experts are saying there are no blocking near term barriers to AGI, that is something to take seriously.

Here are surveys of contributors to major AI conferences.

The y-axis is the probability that ML paper authors gave to reaching AGI, or human-level-machine-intelligence, by each year on the x-axis. Each light line in the background is a single random participant.

In 2022 the population gave a median estimate of AGI by 2060 or so.

In 2023, the median estimate was 2045 or so, with many individuals predicting much shorter timelines.

These surveys are now two years old. I don't know what a survey taken today would show, but I'm betting on a median estimate in the 2030s.

I can also speak to my personal experience. I've mentored a lot of grad students. Currently Claude feels like working with an inept grad student that gets silly things wrong, but also one with encyclopedic knowledge of every field, who is eager and amazingly fast.

This description wasn't true one year ago. It won't be true again one year in the future. The models will be much better than they are today.

https://gco.iarc.fr/overtime/en/dataviz/trends?populations=752&sexes=1_2&types=1&multiple_populations=1&key=asr&age_end=14&cancers=12

)

https://blog.aiimpacts.org/p/2023-ai-survey-of-2778-six-things

https://arxiv.org/pdf/2401.02843

16 of 41

What does it mean for a machine to fly?

Which of these count?

1901: Machine glides 100 meters.
1902: Machine glides 180 meters.
1903: 59 seconds powered flight in straight line before crash.
1904: 5 minutes powered flight, steering in circles.
1905: 38 minutes powered+controlled flight.

Now the precise definition seems irrelevant.�

What does it mean for a machine to be generally intelligent?

There is nuance at the cusp. (c.f. Levels of AGI)
It is AGI when the precise definition seems irrelevant.

17 of 41

I have 2-5 years to contribute intellectually to the world.�What should I do?

Prefer ambitious, targeted, fast collaborations�
The timescale of the project needs to be shorter than the characteristic timescale of the exponential

Otherwise, it's better to wait, and use a newer foundation model to do it better+faster
This does not favor slow, independent, open-ended exploration :(�

The bitter lesson [Sutton, 2019]

General methods that leverage computation are ultimately the most effective.
Work on projects that will benefit from scale.
Avoid projects that will be solved by scale alone, before you even finish the project.

2 years�hard work

If we take these timelines seriously, what should we as individuals do??

These are plausibly our last moments to intellectually shine.

First, we should make sure our projects will still be relevant when they are completed. There is some improving curve of the science that can be done with minimal human effort, just by prompting the best current foundation model. One outcome that you want to avoid is that you work hard for 2 years, and make significant progress, but by the time you're done someone can achieve something better just by asking a foundation model.

This suggests working on targeted projects, with others, so that you can go fast, and stay ahead of the exponential. It discourages slow open-ended exploration.

In 2019 Richard Sutton published an essay called the bitter lesson. He observed that general methods that leverage computation are ultimately the most effective.

The bitter lesson has maybe a positive and negative takeaway. if you work on projects that become more effective as compute and intelligence scales, or even better feed back into that process, then the results will be higher impact than you anticipate.

The negative is that you really, really don't want to work on projects that are going to be solved anyway with scale, possibly before you even finish the project.

People used to spend whole careers and designing features for image classifiers

Choose projects that can productively absorb compute

... and especially projects with feedback effects, where scaling helps with the project, and the project helps with scaling

Choose research that will have multiplicative benefits across scales, rather than research that becomes irrelevant as scale increases.

It also suggests working on projects that benefit from scaling, and feed back into scaling.

Take the bitter lesson seriously.

Go all in, and do something that you will feel proud of afterwards.

You know how sometimes people get halfway through their careers, and then have a crisis of meaning? You don't have time for that shit. Do the meaningful thing now.

Does the bitter lesson apply? Will this project be accomplished by scale alone if we wait?

18 of 41

I have 2-5 years to contribute intellectually to the world.�What should I do?

Learn the new tools.�
They are often non-ergonomic. Use them anyway.

Horseless carriages are awkward, but still outperform carriages with horses�

You should be using foundation models to

Brainstorm + iterate on research ideas
Get writing feedback
Write your code
Iterate on analysis (Claude Code + a VSC notebook works great)
...�

Being a PI is great practice for prompting LLMs, and vice versa.

Requires structured and clear scoping and communication of tasks.

19 of 41

I have 2-5 years to contribute intellectually to the world.�What should I do?

Apply your beliefs to your career (and personal) choices

The job of an academic will be very different in 10-20 years.
Your current career trajectory may not even exist.
There is a reason many academics (✋) are working in industry. If you stay, do something you can only do in academia.�

"I think we're going to have AGI in 3 years. [...] I'm planning to finish my PhD in 2 years, get a postdoc, and apply for faculty jobs after that."�
Risk-reward career tradeoffs just sacrifice reward

There is an unavoidable baseline of AGI risk
Many jobs and institutions are going to be disrupted
Pursue the option with the biggest upside

This talk was motivated by a recent conversation I had with a very capable grad student, where they were sharing a personal timeline for AGI of 3 years ... and then a few minutes later were talking about their career plans, and ["..."]

If you take seriously that AGI is coming in one part of your brain, you should also take it seriously in other parts of your brain.

I don't know what an academic job will look like in a few decades. For that matter, I don't know what any job will look like. But I am sure it will look quite different than it does today. You should not be planning with an assumption that your current career path is a stable one.

People often make life decisions balancing risk vs. reward. If AI is going to cause large scale disruption, and is going to be able to do your job ... then there is a high baseline level of unavoidable risk. This means that if you are choosing between a safer research or career choice, and one with a higher potential upside ... the safer one probably isn't actually safe, and you are mostly just sacrificing upside.

Even without considering AGI, people are often too risk adverse. Do the interesting thing instead of the safe thing.

You are probably calibrated much too conservative.

So when you choose the safe life rather than the one

The safe and stable choice is neither safe nor stable

There is a reason many academics (✋) are working in industry

More leverage on the future

20 of 41

I have 2-5 years to contribute intellectually to the world.�What should I do?

Choose projects that shift the trajectory, rather than introducing a transient change

Do something you will be proud of

After retiring to your villa in the Dyson swarm, you want to be able to tell your grandkids how you helped get the good outcome.

Photo: The geologic epoch after the Anthropocene

if you have a limited time to contribute, and the stakes are high,

you should do something you are proud of!!!

When you are retired in your villa in the Dyson swarm, you will want to feel that you helped get to a positive outcome.

You should also choose projects that will have mattered, looking back. This means you should prefer projects that shift the trajectory post AGI, rather than induce a transient change in the near term.

The photo on the right is the big reveal -- the geologic epoch after the Anthropocene is distinguishable by observing Rock strata, because it has all rock strata converted into a dyson swarm and uniformly distributed around the Sun.

This is a once-in-many-lifetimes chance to shift the trajectory of a historic transformation. These are plausibly your last moments to shine in some particular ways that might be very important to you. Go all in, and do something that you will feel proud of afterwards.

You know how sometimes people get halfway through their careers, and then have a crisis of meaning? You don't have time for that shit. Do the meaningful thing now.

This means doing things that change what other people think or value

[something about dual use?]

21 of 41

You have immense power and leverage over the future of AI

AI can seem like a process beyond our individual control�
This is false�
We are still early in the �exponential

22 of 41

Small choices early in exponential growth have huge consequences

Proposal for standardized email (SMTP) but not chat protocol in 1980
Free sharing of HeLa immortal human cell line
Hobbyist creation of Linux operating system
Introduction of invasive rabbits to Australia as food animal
Work on Haber-Bosch nitrogen fixation process
Choice of lead as anti-knocking agent in gasoline
Proposal of, and choice to commercialize, PageRank algorithm
Founding of professional organization (American Medical Association, American Bar Association, …)
Establishment of academic peer review system

23 of 41

You have immense power and leverage over the future of AI

Individual, small, choices you make will determine the future commercial, social, and political landscape of AI

What research problem? Where do you work? Name model behavior a capability or a risk? Interpretable or inscrutable model architecture? Publicly share an ethical stance? What trajectory/tech tree does your project encourage?�

This is a responsibility as well as a power.�
Be intentional and thoughtful about projects and other choices. The world will change in remarkable ways, and you personally have the leverage to make that change much better or much worse.

24 of 41

Some project areas I like

🩷 AI for science

Often unambiguously positive in expectation

🩷 Science on AI models

Interpretability, statistical physics, neuroscience, psychology, and economics of AI models

🩷 Safety research

Huge importance to field size ratio
Foundations of field -- e.g. the right questions to ask -- are still often missing

🩷 Characterizing (future) capabilities and behaviors

Under-researched: System behavior of many AI agents (+for safety and science on AI)

🩷 Access, equity, fairness

Challenge level: pragmatism on active culture war issues
Fixing UX counts

🩷 Policy and governance

Technical people in policy positions are one of the scarcest resources

I've told you that I think AGI is coming, soon.

And that I think this should motivate you as you make personal and professional and research decisions.

And I've talked about a few specific ways it might feed into that decision making process.

Now here are some particular project areas I like. This list especially is massively incomplete. The most impactful things to do are probably the ones I've left off, and no one is yet thinking about.

If you want to talk in more detail about what work in any of them would look like, please approach me after the talk.

[read list]

AI for science is things like material discovery, protein folding, weather modeling, fusion reactor plasma monitoring, ...

Science on AI models means treating the AI models themselves as the object of study, and using techniques from another field. Better understanding of the systems we are building is also often a positive.

interpretability is maybe the canonical example of this
there is work treating the parameters or activations of neural networks as stat-mech like ensembles
there is work probing the apparanet psychology of AI models
there is work treating the economic behavior and consequences of models

etc. It might be useful to spend an hour brainstorming all the fields that could be used to do science on AI models

There is straight up AI safety research.

This is something you can get in on the ground floor of, which is hugely important.

There's prediction and extrapolation of AI capabilities. The better we understand what the future might look like, the better it will likely go.

There's access, equity, fairness.

If we want this technology to benefit everyone, this is incredibly useful. I also expect it to be a toxic and stressful subfield to work in. It's embracing multiple culture war issues at once, and your work is only going to be net positive if you are able to do that without being dgomatic.

If you are equipped technically and in personality to do work on policy and governance -- DO IT! Governments are desperate for competent technical people to advise them. Everyone good leaves for more pay. But we really need technically competent people informing these policy decisions. This is amazingly high leverage if you do it.

systems both for characterizing, and for safety, and for science on AI models.

For science on AI and AI for science --

These are new fields that didn't exist a few years or even a few months ago.

It doesn't matter if you're not trained in them. No one is trained in them.

And they are going to determine the future shape of society.

You also very much want to be working to automate science, rather than competing against the automation of science.

Access equity inclusion is maybe an extra high challenge level

the right UI maybe pushes out for a few years the period where human can collaborate with AIs to be more productive than the AI on its own.

"Places at the edge of things where interesting things were happening at the intersection between fields and maybe it's riskier and also maybe your colleagues don't recognise the achievement as much because it's unknown territory."

25 of 41

Take the future seriously!

AGI is coming. Consider its implications when deciding what you work on, where you work, when you make career transitions, how you think about the important and interesting problems, how you think about the potential consequences and leverage of your work
Prefer ambitious, targeted, fast collaborations

The timescale of the project needs to be shorter than the characteristic timescale of the exponential

Use the new tools.
Apply your beliefs to your career choices

"I think we're going to have AGI in 3 years. [...] I'm planning to finish my PhD in 2 years, get a postdoc, and apply for faculty jobs after that."
There is no safe option. Try for the best option.

Do something you will be proud of

You have immense power and leverage over the future of AI

This is a good time (the last time?) to go all in

26 of 41

SCRAP

27 of 41

A rubric to decide what to research

Positive impact: if this project works flawlessly, how large is the potential benefit?

Small checkpoints on the way towards large impact are OK.
Derisk early and often!
Timescale of success -- will the impact still matter?

Bitter lesson: will the project be complementary to scale?
Opportunity cost: time and effort and resources
Comparative advantage: why is this the project for you in particular?
Redundancy: how many other people on the planet are doing the same thing?

I promised I would give concrete advice -- so let me share with you an actual rubric I use when choosing research projects. This is slightly off-theme for the rest of the talk. It is partially informed by AGI -- but is a good set of questions to ask about a project regardless of AGI timelines.

First, you should gate by impact. If this project works flawlessly, how large is the potential benefit?

Make sure you evaluate the expected outcome projected onto your value axis, rather than just measuring its norm. See the previous section -- you want to have an impact you can be proud of, and this matters more and more as the potential scale of the impact increases.

This doesn't mean that every project needs to be trying to get a nobel prize.

It's fine to do small projects, but they should be in pursuit of a coherent larger mission or goal. You should be able to tell a story you believe about why they're important.

In fact, doing an ambitious project without breaking it into small steps is a failure mode. If you are going to succeed, you need to be able to break your ambitious project into small quantifiable steps.

In a fast changing field, you should evaluate the impact the project will have when it is complete, not the impact it would have if it appeared now.

28 of 41

A rubric to decide what to research

Positive impact: if this project works flawlessly, how large is the potential benefit?
Bitter lesson: will the project be complementary to scale?

Your work should still matter after intelligence is too cheap to meter
The impact of the right project can grow exponentially in time
data, best practices and frameworks, new questions, new framings, algorithms that work better with more scale

Opportunity cost: time and effort and resources
Comparative advantage: why is this the project for you in particular?
Redundancy: how many other people on the planet are doing the same thing?

29 of 41

A rubric to decide what to research

Positive impact: if this project works flawlessly, how large is the potential benefit?
Bitter lesson: will the project be complementary to scale?
Opportunity cost: time and effort and resources

Is the effort reusable?
(again) Derisk early and often!

Comparative advantage: why is this the project for you in particular?
Redundancy: how many other people on the planet are doing the same thing?

30 of 41

A rubric to decide what to research

Positive impact: if this project works flawlessly, how large is the potential benefit?
Bitter lesson: will the project be complementary to scale?
Opportunity cost: time and effort and resources
Comparative advantage: why is this the project for you in particular?

specific expertise
access to resources (e.g. compute, data)
collaborators with the right expertise
I have a clever idea no one else has thought of

Redundancy: how many other people on the planet are doing the same thing?

31 of 41

A rubric to decide what to research

Positive impact: if this project works flawlessly, how large is the potential benefit?
Bitter lesson: will the project be complementary to scale?
Opportunity cost: time and effort and resources
Comparative advantage: why is this the project for you in particular?
Redundancy: how many other people on the planet are doing the same thing?

Even if you publish first, your time was wasted! The world should be in a different state because of your efforts.
Find ideas that no one else recognizes as important

Redundancy: How many other people on the planet are presently trying to solve the same problem in roughly the same way? Even if you win the publication race, it is wasted effort if the same thing would have been discovered at roughly the same time with or without your work.

You want the world to be in a different state because of your efforts.

This is the one that I see people fail at the most often. If everyone agrees that you are working on a very important problem, then you shouldn't be working on it. Someone else will do it. You want to get on the wave before it's crashing, not afterwards.

This is also the one that applies differently in industry than in research. If you're running a company, often you need to try things that other companies are also doing.

--

So to use this, you should explicitly score each of these. Then of course you should throw the scores away and go with your gut. But the process of thinking this through can be hugely beneficial.

32 of 41

A rubric to decide what to research

Positive impact: if this project works flawlessly, how large is the potential benefit?
Bitter lesson: will the project be complementary to scale?
Opportunity cost: time and effort and resources
Comparative advantage: why is this the project for you in particular?
Redundancy: how many other people on the planet are doing the same thing?

33 of 41

A rubric to decide what to research

What is the absolute weirdest project (research or non-research) you would enjoy doing? The sweet spot:

You can clearly articulate why the project is a good idea
Everyone looks at you funny when you first explain it

Positive impact: if this project works flawlessly, how large is the potential benefit?
Bitter lesson: will the project be complementary to scale?
Opportunity cost: time and effort and resources
Comparative advantage: why is this the project for you in particular?
Redundancy: how many other people on the planet are doing the same thing?

Also, I just want to say -- people never go weird enough!!!

Whatever project you are working on, you should choose a weirder one.

You are going to be judged on the best things you accomplish, not on the typical thing you accomplish.

This means you should be optimizing for the long tail of surprising very positive outcomes.

Try something that is weird, but might be amazing, rather than something that is probably OK.

The ideal thing to work on is something where you can clearly explain why it is a good idea, but when you explain it to other people they look at you funny and have trouble getting it. This is the strongest signal of future project success!! You need the first part too though -- you need to at least be able to clearly and concretely explain to yourself why it is a good idea.

34 of 41

Should you do academic research?

What are you optimizing for?

Love of science? (Scientific) fame? Respect? Money? Interesting problems? Amazing friends and colleagues? Positive impact on world?

If this was your last job, would you do something different?

Are you in a holding pattern for a future career?

What would you do instead? What is your best alternative?

startup, industry research, industry engineering, advocacy, family, ...?

How much leverage over the future will you have in each case?

So that was a rubric for how to choose an academic research project.

There is a second question, which is hinted at in this talk, which is should you do academic research? There is not a right answer to this, but it is worth thinking through in a structured way.

I can share where I landed on this question, which is that I have mostly stopped doing academic research.

I especially miss being able to tell people about my work.

But there are benefits -- I have more compute, and money, and access to different cutting edge problems, than I think I would have in academia. I also arguably have more leverage on the future.

In a parallel universe, I am running a small academic lab, and loving it. In many ways that is a more appealing life path to me. But I believe we are building a technology that will utterly change the world, and I can't stay on the sidelines for that.

The biggest deciding factor is what values you are optimizing for in your career. [read]

The future is unknowable, but this may also be your last job you contribute to intellectually. If it was, is it the job you want?

You should make sure you are getting as much value out of the job now as you can -- it should be valuable in its own right, not because it gets you where you hope to go next.

It also matters whether any of the alternatives appeal to you.

For me, it matters how much I feel like I will matter. I want to do something where I feel like I have leverage on the future. It's not obvious where that is -- it very much depends on the project you choose.

--

mediocre harvard researchers can become amazingly effective, when they switch to a different pond.

if you want to do work that matters, you need to do things that other people don't realize are important yet.

you will also be judged by your best output, not your typical or worst output.

so this incentivizes going for the weirdest project you can find, that only you realize is important. If you have a project where you can clearly verbalize why it's important, but also when you explain it to people they look at you a little funny -- that is the project you want to be doing!!

35 of 41

[Andy Jones, personal communication, 2025]

36 of 41

Cancer deaths are exponentially falling,�in the rich world

37 of 41

38 of 41

[Wiedemer et al., 2025]

39 of 41

40 of 41

41 of 41

I have 2-5 years to contribute intellectually to the world.�What should I do?

Apply your beliefs to your career (and personal) choices

The job of an academic will be very different in 10-20 years.
Your current career trajectory may not even exist.�

"I think we're going to have AGI in 3 years. [...] I'm planning to finish my PhD in 2 years, get a postdoc, and apply for faculty jobs after that."�
There is a reason many academics (✋) are working in industry

More leverage on the future
More access to cutting edge problems
More resources
(More money)

This talk was motivated by a recent conversation I had with a very capable grad student, where they were sharing a personal timeline for AGI of 3 years ... and then a few minutes later were talking about their career plans, and ...

If you take seriously that AGI is coming in one part of your brain, you should also take it seriously in other parts of your brain.

I don't know what an academic job will look like in a few decades. For that matter, I don't know what any job will look like. But I am sure it will look quite different than it does today. You should not be planning with an assumption that your current career path is a stable one.

This is a complicated tradeoff -- but there are also reasons that industry is particularly appealling at this point in time. [bullets]

In a parallel universe, I am running a small academic lab, and loving it. In many ways that is a more appealing life path to me. But I believe we are building a technology that will utterly change the world, and I can't stay on the sidelines for that.

These are plausibly your last moments to shine. Go all in, and do something that you will feel proud of afterwards.

You know how sometimes people get halfway through their careers, and then have a crisis of meaning? You don't have time for that shit. Do the meaningful thing now.