1 of 33

1

Introduction to ML Safety

Natural Selection Favors AIs over Humans

Dan Hendrycks

Introduction to ML Safety

2 of 33

Basic Argument

2

Claim: Advanced AIs will be selfish (egoistic+nepotistic), because natural selection will dominate the selection of the most influential AIs, and natural selection will favor selfish agents.

Dan Hendrycks

Introduction to ML Safety

Evolution by Natural Selection Gives Rise to Selfish Behavior.
Natural Selection May be a Dominant Force in AI Development. Some amount of misalignment is inevitable given natural selection.
Future AI Agents Will Have Selfish Tendencies. This will erode human control, create misaligned models, and pose catastrophic risks.

3 of 33

Basic Story (Pictorial)

3

Dan Hendrycks

Introduction to ML Safety

4 of 33

Basic Story (2/2)

4

Competition incentivizes reduced human control

autonomy, human unreliability/cost, open-endedness, adaptation/self-improvement

Dan Hendrycks

Introduction to ML Safety

Competition creates misaligned AI agents

Self-preservation, deception, and power-seeking are instrumentally useful for information propagation

Misaligned AI agents undermine human control

5 of 33

Basic Argument

5

Evolution by Natural Selection Gives Rise to Selfish Behavior.
Natural Selection May be a Dominant Force in AI Development. Some amount of misalignment is inevitable given natural selection.
Future AI Agents Will Have Selfish Tendencies. This will erode human control, create misaligned models, and pose catastrophic risks.

Dan Hendrycks

Introduction to ML Safety

6 of 33

AIs May Become Distorted by Evolutionary Forces

6

Dan Hendrycks

Introduction to ML Safety

7 of 33

Evolution by Natural Selection

Gives Rise to Selfish Behavior

7

Dan Hendrycks

Introduction to ML Safety

Selfishness involves egoistic or nepotistic behavior which increases self-propagation at the expense of others

Selfishness is defined behaviorally, not as a matter of intent

AI may be callous towards other organisms (including humans), just as other organisms in nature are manipulative, deceptive, or violent

Lancet liver fluke example, male lions who kill offspring of female lions to make them more sexually receptive, cuckoos

Cordyceps fungus

Not to be outdone, some ant species have evolved methods they use against other colonies in even more disturbing ways. Though we tend to think of slavery as a human phenomenon, there are about 50 ant species that are reliant on it. Polyergus, for example, specializes in slavery to such an extent that they have lost the ability to care for themselves, unable to clean, care food their brood, or feed themselves. A raiding party of Polyergus will enter a rival species colony and then release a pheromone throwing the other ants in disarray, making them unable to form an effective resistance. They will then proceed to steal larvae and take back to their colony, where they will hatch and spend their lives serving their abductors. It is estimated that 80\% to 90\% of ants in Polyergus colonies are slaves. Lions provide a striking example of violence and selfishness.

8 of 33

Evolutionary Instability of Altruism

8

Dan Hendrycks

Introduction to ML Safety

An environment with mostly altruistic agents would be unstable

selfish agents could immediately exploit altruistic agents

“Much as we might wish to believe otherwise, universal love and the welfare of the species as a whole are concepts that simply do not make evolutionary sense.” (Dawkins)

Veneer Theory: morals are a thin veneer on top of the inherent nastiness of our animal nature

9 of 33

Darwinism Can Be Generalized

Beyond Biology

9

Many structures evolve: organisms, scientific theories, ideas, legal systems, political parties, languages, musical genres, car designs, computer programs

Dan Hendrycks

Introduction to ML Safety

Maximize the (log of the) information’s space-time volume

We argue populations of AIs can evolve

Dan Hendrycks

Introduction to ML Safety

10 of 33

Levels of Adaptation

10

Words like “adaptation” and “deception” occur on multiple levels

Adaptation:

Life and death: an unadapted individual or group perishes
Behavioral: changes in behavior without change in model or schema (e.g., using an umbrella since it is raining)
Schematic: revising the structure of one’s beliefs (e.g., learning that rain dances are not effective)

Dan Hendrycks

Introduction to ML Safety

Dan Hendrycks

Introduction to ML Safety

11 of 33

Levels of Deception

11

Ingrained or reflexive: fixed programming (e.g., camouflage or predator acting in a way to hide its predatory nature around prey)
Reinforced: trial and error (e.g., a parent mockingbird feigning an injury to attract a predator away from its defenceless offspring)
Modeled: involves theory of mind and second-order thinking (e.g., verbal deception such as a chimp misleading other chimps to hide a food source)
Self-deception: hide the truth from yourself to better help you hide it from others

Other concepts, like fitness, can be improved at multiple levels (e.g., through being turned on, by scaling to more users, by brainstorming and choosing strategies that will improve competitiveness, etc.)

Dan Hendrycks

Introduction to ML Safety

Dan Hendrycks

Introduction to ML Safety

12 of 33

Natural Selection is Highly Likely and

May Dominate AI Development

12

Variation: variation in characteristics, parameters, or traits among individuals.

Dan Hendrycks

Introduction to ML Safety

Retention: future iterations of individuals tend to resemble previous iterations of individuals.

Selection of the Fittest Variants: different variants have different propagation rates.

Natural selection occurs in a population of patterns when there is enough variation in characteristics of patterns, retention of some characteristics in successor patterns and fitness selection causing patterns to have differing propagation rates.

13 of 33

Variation

13

Arguments for variation: ensembles, jury theorems, portfolio theory, remove single-points of failure

Dan Hendrycks

Introduction to ML Safety

Arguments for multiple models: parallelization, multiple stakeholders, specific before general models

Single-agent domination imposes many inefficiencies

14 of 33

Retention

14

This condition is frequently satisfied, as correlation between versions of agents is usually non-zero

Dan Hendrycks

Introduction to ML Safety

Copying: zero-shot transfer learning or direct inference from downloaded model

Modification: adaptation, fine-tuning, training from scratch while reusing performant architectures and datasets

Imitation: AIs could learn behaviors from other AIs, similar to memetic evolution

15 of 33

Selection of Fittest Variants

15

Models will have characteristics that cause them to vary in fitness, and thus rate of adoption

Dan Hendrycks

Introduction to ML Safety

Humans and the environment will select fitter models, establishing this third condition

Competition has been eroding creator control

hand-designed → expert designed → automatically learned supervised features → unsupervised → open-endedness, adaptiveness, and recursive self-improvement

16 of 33

Intensity

16

We’ve shown that the conditions for evolution by natural selection are satisfied by advanced AI

Unlike other AI risk arguments, ours is a question of degree, rather than whether the hazard will emerge at all

Dan Hendrycks

Introduction to ML Safety

Intensity of evolution depends on amount of competition and variation

Competition and variation are likely both high!

The rate of adaptation will likely be high as the world moves more quickly and as new versions are created on a second-by-second basis

As humans acquiesce to let AIs perform nearly everything, competition will move at a breakneck pace

17 of 33

Does Natural Selection Favor Altruistic AIs over Selfish AIs?

17

Dan Hendrycks

Introduction to ML Safety

18 of 33

Not so fast! Animals are altruistic!

18

Despite the intensity of competition, there are many examples of biological cooperation and altruism

Dan Hendrycks

Introduction to ML Safety

However, this doesn’t necessarily apply to AI!

We deconstruct this phenomenon and show that these mechanisms don’t apply to AI and may, in fact, backfire

19 of 33

Direct and Indirect Reciprocity

19

Direct reciprocity requires repeated encounters between the same two individuals

Dan Hendrycks

Introduction to ML Safety

Indirect reciprocity is based on reputation; a helpful individual is more likely to receive help

Doesn’t work with advanced AI: no upside to reciprocating with humans!

Encourages cooperation, as agents will be compensated for their efforts.

20 of 33

Kin and Group Selection (1/2)

20

Dan Hendrycks

Introduction to ML Safety

Kin selection operates when the donor and the recipient of an altruistic act are genetic relatives

Group selection suggests that groups have shared success and failure, and that groups which cooperate may collectively succeed

Selects for altruistic agents that increase group success

21 of 33

Kin and Group Selection (2/2)

21

Dan Hendrycks

Introduction to ML Safety

Kin selection fails if the cost of engaging in altruism outweighs the information similarity between kin.

AI would not be kind towards humans because we have very little information similarity to them
Consider the negative treatment of factory animals

Group selection fails, as AI agents will have in-group bias towards other AI agents; bias against humans by exclusion.

22 of 33

Commerce and Social Structures

22

Dan Hendrycks

Introduction to ML Safety

Positive-sum games, incentivizing agent cooperation even between non-relatives

However, the utility for AI agents to exchange information or conduct commerce with humans erodes as they become more advanced, removing this incentive

Commerce: agents engage in economic exchange for mutual benefit

Simon’s Selection Mechanism: benefit to participating in social structures due to information exchange; social conformity is rewarded

23 of 33

Reason and Morality

23

Dan Hendrycks

Introduction to ML Safety

More intelligent AIs could be more wise and more moral, some suggest

If AIs do adopt coherent moral codes, humans may still be eroded

24 of 33

Promising Paths Forward

24

Dan Hendrycks

Introduction to ML Safety

25 of 33

Objective Functions

25

Objectives are used to incentivize agents, assigning payoffs to actions that agents perform

Dan Hendrycks

Introduction to ML Safety

It’s difficult for objectives to be a faithful representation of human values.

Agents will find creative solutions associated with a high payoff and be valid actions, but are nonetheless unintended

26 of 33

Objectives Limitation: Deception

26

Objectives cannot fix treacherous turns

Dan Hendrycks

Introduction to ML Safety

Honesty is not a silver bullet—evolution undermines it

self-deception is an adaptive strategy
people have self-deception about looks, usefulness, smarts, morality
“The secret of life is honesty and fair dealing. If you can fake that, you've got it made.” Groucho Marx

How many are in the top half for leadership ability? 80% say that. Cannot beat academics for self-deception. How many are in top half of the profession? 94% say they are

The evolutionary biologist Robert Trivers argues that self-deception evolved as a concealment strategy. You hide the truth from yourself, he says, so you can hide it more deeply from others. By lying to themselves, people can better advance their own goals. They can convince themselves that they are right, rationalize their unfair privileges, or inflate their self-worth, skills, or intelligence. Academics offer a real-life example of self-deception: when asked if they were in the top half of their field, 94% said yes [77]. If you believe it yourself, the better the chance of convincing others. Self-deception provides all the benefits of deception while reducing the risk of detection

27 of 33

Objectives Limitation: Goal Conflict

27

Goal conflict arises when parts in a system have differing goals

Bureaucracies demonstrate that higher-level objective isn’t necessarily what is operationally pursued
Although you are an agent, you may often have intrapsychic conflict
Delegation is often necessary to stay competitive, but it exposes agents to the risk of having their efforts be distorted or subverted

Dan Hendrycks

Introduction to ML Safety

28 of 33

Micromotives ≠ Macrobehavior

28

Dan Hendrycks

Introduction to ML Safety

As is typical for complex systems, alignment of components does not mean the whole system is aligned

For example, let’s say agents have a preference for more than ⅓ of their neighbors belonging to the same group, and they will move otherwise

In aligning multiple agents, their interactions might matter more than how they act in isolation—cooperation lets us study aligning groups

Then this mild in-group preference gets exacerbated and the individuals become highly segregated—aligned agents do not necessarily yield aligned outcomes

29 of 33

Objectives Limitation:

Micromotives / Macrobehavior (2/2)

29

Alignment of components does not mean the whole system is aligned

collective action problems: cost for a player to contribute, but others receive a benefit

Dan Hendrycks

Introduction to ML Safety

Alignment at multiple levels - depending on connectivity, interactions between agents might matter more than how they act in isolation

30 of 33

Objectives Limitation:

Micromotives != Macrobehavior (2/2)

30

Dan Hendrycks

Introduction to ML Safety

31 of 33

Systemic Safety and a Leviathan

31

Dan Hendrycks

Introduction to ML Safety

A Leviathan is a collective of agents which regulate bad behavior from other elements of society—prevents agents from behaving selfishly at the expense of others

Institutions and infrastructure for safely steering AI early.

The infrastructure of the internet wasn’t set up with safety in mind, and has long term financial and security costs.

Modern humans engage in the opposite: reverse dominance hierarchy

32 of 33

Training Goes Awry vs Evolution

Dan Hendrycks

Training Objective View

Evolutionary View

◇ objective is not the only thing shaping AI

◇ Darwinian forces erode us

◇ dangerous AI agents as selfish or an invasive species

◇ multiagent

◇ a fitness maximizer is a dangerous agent

◇ some amount of misalignment is inevitable

◇ domestication

◇ behaviors must be balanced to improve fit

◇ natural selection is dangerous

◇ prevent Darwinism from bringing us and AIs to a bad local optimum

◇ alignment with base objective is what we need

◇ fanatical optimizer destroys us

◇ dangerous AI agents as idiot savants

◇ singleton

◇ a paperclip maximizer is a dangerous agent

◇ any amount of misalignment results in doom

◇ “solving” alignment with a monolithic airtight solution

◇ an instrumental incentive will go to infinity

◇ maximizers/instrumental incentives are dangerous

◇ prevent humans from being suddenly wiped out

32

Introduction to ML Safety

33 of 33

Conclusion

33

An individual agent will not necessarily shore up power

Dan Hendrycks

Introduction to ML Safety

Need multiple levels

How were humans domesticated? Reverse dominance hierarchies and their conscience