1 of 28

Introduction to Existential Risks

Mitigating the greatest risks to the future of humanity

2 of 28

HUMANITY’S POTENTIAL

3 of 28

EXISTENTIAL RISK

4 of 28

EXISTENTIAL RISK

5 of 28

EXISTENTIAL RISK

The destruction of humanity’s longterm potential

6 of 28

NATURAL

Asteroids

Comets

Supervolcanic Eruptions

Supernovae

Gamma Ray Bursts

ANTHROPOGENIC

Nuclear War

Climate Change

Environmental Damage

EMERGING

Engineered Pandemics

Unaligned AI

Dystopian Scenarios

7 of 28

NATURAL

Asteroids

Comets

Supervolcanic Eruptions

Supernovae

Gamma Ray Bursts

1 in 10,000

ANTHROPOGENIC

Nuclear War

Climate Change

Environmental Damage

EMERGING

Engineered Pandemics

Unaligned AI

Dystopian Scenarios

8 of 28

NATURAL

Asteroids

Comets

Supervolcanic Eruptions

Supernovae

Gamma Ray Bursts

1 in 10,000

ANTHROPOGENIC

Nuclear War

Climate Change

Environmental Damage

1 in 300

EMERGING

Engineered Pandemics

Unaligned AI

Dystopian Scenarios

×30

9 of 28

NATURAL

Asteroids

Comets

Supervolcanic Eruptions

Supernovae

Gamma Ray Bursts

1 in 10,000

ANTHROPOGENIC

Nuclear War

Climate Change

Environmental Damage

1 in 300

EMERGING

Engineered Pandemics

Unaligned AI

Dystopian Scenarios

1 in 6

×30

×50

10 of 28

This largely comes from risks from synthetic biology, which could be used to engineer highly viral and lethal pathogens and cause global pandemics, and risks for uncontrolled transformative artificial intelligence, or artificial intelligence with sufficient capabilities and societal use to transform society in a manner akin to the industrial revolution, if developed too hastily, without sufficient attention paid to safety, and ensuring that the goals it pursues benefit all of humanity.

Surprisingly to me, Ord estimates that most of the risk comes from transformative AI and engineered pandemics, not climate change and nuclear war, which were the first risks that came to mind for me when I thought about extinction. Ord estimates that these guesses are in the correct order of magnitude (or that there’s a decent chance he’s off by a factor of 3), so that these estimates shouldn’t be taken too literally, but show the general order of magnitudes and ranking between risks.

Now I want to dig a little deeper into what makes an existential catastrophe so bad — and why we need to strive so hard to avoid them. In some ways, this may seem obvious. Of course the destruction of our future would be bad! But it is less obvious just how profoundly bad it would be, or how robust this case is. And we need to understand this in order to know how much to prioritise existential risk among all the issues of our time, especially as many of the things we could do will not directly make the difference between survival and failure, but will instead improve our chances by a modest amount.

11 of 28

PRESENT

PAST

FUTURE

COSMIC SIGNIFICANCE

12 of 28

THE PRESENT

The billions of lives destroyed in the downfall

The most obvious reasons are rooted in the present.

It is technically possible to have an existential catastrophe that does relatively little harm to the people alive at the time. For example, if we all decided to have no more children, that would destroy our future but may (let us suppose) cause no other suffering. But very few risks are of this kind. The downfall of humanity, or civilization, would involve untold suffering and loss, as billions of people were killed in the catastrophe. Or perhaps instead oppressed by a tyrannical regime.

The moral case for preventing such horror requires little elaboration.

We have seen such losses on smaller scales: with thousands or millions of lives being destroyed. And other things being equal, millions of deaths must be much worse than thousands; billions much worse than millions. Even though we lose our ability to fully comprehend these losses, the numbers offer us a guide to the moral stakes. Even measured just in terms of loss of life or immediate suffering, most kinds of existential catastrophe would easily be the worst event in humanity’s long history.

13 of 28

THE PAST

The partnership of the generations
Completing intergenerational projects
Paying forward our debt
Preservation & stewardship
Making things right

But this isn’t the only way to make a case for the pivotal importance of existential risk. Consider our relationship to the past.

We are not the first generation. Our cultures, institutions, and norms; our knowledge, technology, and prosperity; were gradually built up by our ancestors, over the course of ten thousand generations. Humanity’s remarkable success has been entirely reliant on our capacity for intergenerational cooperation. Without it we would have no houses or farms, we would have no traditions of dance or song, no writing, no nations.

Edmund Burke:

“It is a partnership in all science; a partnership in all art; a partnership in every virtue, and in all perfection. As the ends of such a partnership cannot be obtained except in many generations, it becomes a partnership not only between those who are living, but between those who are living, those who are dead, and those who are to be born.“

And Toby Ord in response, says: Indeed, when I think of the unbroken chain of generations leading to our time and of everything they have built for us, I am humbled. I am overwhelmed with gratitude; shocked by the enormity of the inheritance and at the impossibility of returning even the smallest fraction of the favour. Because a hundred billion of the people to whom I owe everything are gone forever, and because what they created is so much larger than my life, than my entire generation.

This might give us reasons to safeguard humanity that are grounded in our past—obligations to our grandparents, as well as our grandchildren.

If we were to drop the baton—succumbing to an existential catastrophe—we would fail our ancestors in many different ways.

• We would fail to achieve the dreams they hoped for, as they worked towards building a just world;

• We would betray the trust they placed in us, their heirs, to preserve and pass on their legacy

• And we would fail in any duty we had to pay forward the work they did for us, to help the next generation as they helped ours.

Moreover, we would lose everything of value from the past that we might have reason to preserve. Extinction would bring with it the ruin of every cathedral and temple, the erasure of every poem in every tongue, the final and permanent destruction of every cultural tradition the Earth has known. In the face of serious threats of extinction, or of a permanent collapse of civilisation, a tradition rooted in preserving or cherishing the richness of humanity would also cry out for action.

Finally, we might have duties to the future arising from the flaws of the past. For we might be able to make up for some of our past wrongs. If we failed now, we could never fulfill any duties we might have to repair the damage we have done to the Earth’s environment—cleaning up our pollution and waste; restoring the climate to its pre-industrial state; returning ecosystems to their vanished glory.

And this brings us to the future-focused case for mitigating existential risk:

14 of 28

THE FUTURE

The loss of all forms of value across the future

Flourishing, beauty, culture, discovery, achievement

Longtermism

A Primary determinant of ethical value = value for long-term future
The quality of life that future persons may experience, or the grand civilizations that they may build
Reasons:

Future people are morally important
The number of people that could exist in the future

But an existential risk doesn’t just destroy our present, it destroys our future.

The philosopher Derek Parfit had a thought experiment that teases out this difference.

…

The war that kills 100% of people doesn’t just kill humans, it kills *humanity*.

Our future could be so much longer and better than our fleeting present, and there are actions that only our generation can take that would affect that entire span of time. This value of our future could be understood in terms of all the value in all the lives in every future generation. Or in many other terms. Because, in expectation, almost all of humanity’s life lies in the future, so almost everything of value lies in the future as well: almost all the flourishing; almost all the beauty; our greatest achievements; our most just societies; our most profound discoveries. This is our potential—what we could achieve if we pass the Precipice and continue striving for a better world.

This is the longtermist argument for safeguarding humanity’s potential. Longtermism is the idea that a primary determinant of the moral value of an action is its impact over the long-term future. There are two main reasons that motivate this belief: First, the idea that future people are morally important - that the circumstances of someone’s birth, including when they happen to exist, much like where they’re from or their gender, race, or other features, should not influence their moral importance. And second, the number of people that could exist in the future is incredibly vast.

15 of 28

GPI - University of Oxford. YouTube (2020).

16 of 28

GPI - University of Oxford. YouTube (2020).

17 of 28

future

GPI - University of Oxford. YouTube (2020).

18 of 28

COSMIC SIGNIFICANCE

The only moral agents?
The only beings who can understand the universe?
The only place with life?

One common intuition in response to thinking about the size of the future is that maybe there should be some discounting of future lives, as is done in economics for discounting money in the future for reasons like uncertainty, interest rates and inflation making the same amount of money less valuable in the future, hyperbolic discounting and caring about short-term effects mroe than long-term effects personally, and others.

William Nordhaus, is an American economist who, with Paul Romer, was awarded the 2018 Nobel Prize for Economics for his contributions to the study of long-term economic growth and its relation to climate change, and suggested a discount rate of 1.5% per year, which led to an estimate of the social costs of a ton of carbon to be $16.

Does 1.5% seem reasonable? Too low? Too high?

19 of 28

COSMIC SIGNIFICANCE

The only moral agents?
The only beings who can understand the universe?
The only place with life?

20 of 28

COSMIC SIGNIFICANCE

The only moral agents?
The only beings who can understand the universe?
The only place with life?

Whether we are alone in the universe is one of the greatest remaining mysteries of science. Eminent astronomers such as Martin Rees, Max Tegmark and Carl Sagan have reasoned that if we are alone, our survival and our actions might take on a cosmic significance. While we are certainly smaller than the galaxies and stars, less spectacular than supernovae or black holes, we may yet be one of the most rare and precious parts of the cosmos. The nature of such significance would depend on the ways in which we are unique.

If we are the only moral agents that will ever arise in our uni- verse—the only beings capable of making choices on the grounds of what is right and wrong—then responsibility for the history of the universe is entirely on us. This is the only chance ever to shape the universe toward what is right, what is just, what is best for all. If we fail, then the potential not just of humanity, but of all moral action, will have been irrevocably squandered.

Alternatively, if we are the only beings capable of wondering about the universe, then we might have additional reason to seek such understanding. For it would only be through us that a part of the universe could come to fully understand the laws that govern the whole.

And if Earth is the only place in the universe that will give rise to life, then all life on Earth would have a key significance. Earth would be the only place where there was so much complexity in each drop of water, the only place where anything lived and died, the only place where anything felt, or thought, or loved. And humanity would be the only form of life capable of stewarding life itself, protecting it from natural catastrophes and, eventually, taking it to flourish throughout the cosmos.

21 of 28

PRESENT

PAST

FUTURE

COSMIC SIGNIFICANCE

22 of 28

NEGLECTEDNESS

Not only important, but neglected
An intergenerational global public good
Short timescale of politics
The lack of an active constituency
Too big
Very recent

Despite this, existential risks are hugely neglected. To provide some examples, the annual budget of the Biological Weapons Convention, the main international body responsible for keeping states from developing biological weapons, which could either accidentally or purposefully cause regional or global pandemics, is around 1.4 million dollars, which is less than the average McDonalds restaurant. There are only around 100 researchers around the world looking into long-term risks from transformative AI and how to mitigate existential catastrophes from AI. There are only two published climate models of nuclear winter since 1991, while hundreds of missiles remain on high alert, minutes away from being fired.

And there is tremendous scientific work on climate change, but extremely little on scenarios where there could be six or more degrees of warming. The world spends more on ice cream than it does preventing existential risk every year.

23 of 28

NEGLECTEDNESS

Not only important, but neglected
An intergenerational global public good
Short timescale of politics
The lack of an active constituency
Too big
Very recent

24 of 28

NEGLECTEDNESS

Not only important, but neglected
An intergenerational global public good
Short timescale of politics
The lack of an active constituency
Too big
Very recent

25 of 28

NEGLECTEDNESS

Not only important, but neglected
An intergenerational global public good
Short timescale of politics
The lack of an active constituency
Too big
Very recent

26 of 28

~1 in 6

What do we do?

Ord’s estimated risk of human extinction within a century

27 of 28

Tackling existential risks with your career

28 of 28

Books