Singularity Summit 2012

The Singularity, Promise and Peril


For more transcripts, videos and audio of Singularity Summit talks visit

Speaker: Luke Muehlhauser

Transcriber(s): Ethan Dickinson and John Maxwell

Host: Luke Muehlhauser is next, he is the Executive Director at the Singularity Institute. He has published dozens of articles on technological forecasting, intelligence explosion theory, and the cognitive science of rationality, including the online book "Facing the Singularity." Previously, he interviewed dozens of scientists and philosophers for his podcast, "Conversations from the Pale Blue Dot," and has taught classes for the Center for Applied Rationality. He is currently developing several papers, including a survey of proposals for dealing with superhuman artificial intelligence, and a survey of recent progress in the field of AI. Please join me in welcoming the Executive Director of the Singularity Institute, Luke Muehlhauser.

Luke Muehlhauser: Wow, it's great to see you all here. I think this is the largest audience that I've given a talk to before. I'm a little nervous, so I want to let you know that you all look great in your underwear. [laughter] Clearly I'm not a comedian, so let me talk about AI or nanotech or something.

My talk today is "Superhuman AI: Promise and Peril." My talk is an introduction to the risks and opportunities associated with superhuman artificial intelligence, and so for some of you who are veterans of singularity thought, I may not hear any new ideas for you, but hopefully it's entertaining anyway; but I think for most of you there will be some new ideas in here, and in any case, these are the most important points I can think of to talk about with anyone.

Let me start here. Andrew McAfee is a management scientist at MIT, and he co-wrote the excellent little book "Race Against the Machine," and he likes to go around asking people the question, "What are the most important developments in human history?" Some people name important historical figures, other people name empires, some name intellectual breakthroughs in math and sciences and the arts.

How do we tell which of these is actually the most important? Well, we can collect the data on how many people there were at a given time and how well-off they were. When we chart those curves on the screen here, we should see that the actually important events among these should bend those curves in some way. Before I plot the curves, you can all think to yourselves, what do you think are the important events here, which ones are going to bend the curves? This is a chance to test your beliefs against reality.

It was a trick question. None of these events mattered at all. Not even cultural differences, the East and West look the same. The only thing that mattered was technology. Technology is what reshapes our world, it's the most important thing we can think or talk about.

That's going to be the subject of my talk. You already know that technology is important, you came to the Singularity Summit. So I'm going to talk about the most important technology that will ever be invented, and that's superhuman AI.

Why is superhuman AI the most important technology that will ever be invented? It's because intelligence is very powerful. In fact, intelligence is a lot like magic. In the ancient myths of the world, a person with magical powers could kill someone instantly, or fly off into the sky, or communicate with someone on the other side of the world. And intelligence is a lot like magic, except that it's real. If you have enough intelligence, you can figure out how to give yourself whatever other powers you want, including more intelligence.

Or, the power of flight. We actually did invent jetpacks. You won't find them at your local Honda dealer because they're very expensive and highly, highly fatal. [laughter] But we did invent them. I would have loved to fly in here on a jetpack, but the death and the screaming and the fire and stuff probably wouldn't be good.

Arthur C. Clarke once said, "Any sufficiently advanced technology is indistinguishable from magic." That's true, but where does that technology come from? It comes from intelligence. So we might as well say, "Any sufficiently advanced intelligence is indistinguishable from magic."

Now, I don't get to say this next line very often, but: Glenn Beck has some intelligent things to say about this. [laughter] Note to the other speakers I think the audience is about 30-70 on Glenn Beck, good to know. In his latest book "Cowards", Glenn Beck says, "Consider the difference between humans and chimpanzees, which share 95% of their genetic code. A relatively small difference in intelligence gave humans the ability to invent farming, writing, science, democracy, capitalism, birth control, vaccines, space travel, and iPhones, all while chimpanzees kept flinging poo at each other." [laughter] To a chimpanzee, the iPhone or the space shuttle, they're just magic.

So intelligence is powerful. A small difference in brain architecture and suddenly cities are springing up on Earth, and footprints appear on the Moon. And human intelligence is pretty cool, but let's be honest. There's room for improvement. [image on screen of man with ballcap turned backwards, shielding his eyes from the sun with his hand, with the caption "HAT FAIL"] [laughter]

I like seeing the waves of different people getting it. [laughs] [laughter]

And I don't just mean that there's room for improvement from the bottom rung of the species. Einstein might seem vastly more intelligent than a guy who doesn't know how to use a hat, but the difference between the two of them is actually much smaller than the difference between the "hat fail" guy and a mouse, or between Einstein and a superintelligent artificial intelligence.

Really, it's not fair, Einstein doesn't stand a chance against a superhuman AI. Einstein's intelligence runs on sluggish neurons, the AI can fire signals at light speed. Einstein's brain is trapped in a tiny skull, the AI's brain can be as big as a warehouse, or a city. If we discover new algorithms that work better than the ones that evolution gave us, Einstein doesn't get a chance to rewrite his own brain to use those algorithms instead of the ones that evolution came up with, but if an AI discovers these algorithms, then it can rewrite its own code and become qualitatively smarter. And so on.

With these kinds of advantages, a superhuman AI can be just incomprehensibly more effective at achieving its goals than Albert Einstein or any human can be. It can be incomprehensibly better at science, at music, at social manipulation... these are all things done with the mind, not the kidneys.

But some people wonder, "Can we really create superhuman AI?" I'd like to get some historical perspective on this. It's definitely true that some early predictions about AI progress were naive, but at the same time, one obstacle after another has been conquered.

In 1954, it looked to some like machines would never do probabilistic reasoning, but a few decades later probabilistic induction was the standard paradigm in AI. In 1965 the philosopher Hubert Dreyfus mocked the entire field of AI as "alchemy," and he mocked the ability of 1965 chess programs, chess programs which defeated him two years later. [laughter] In 1973, Satosi Watanabe predicted that AI would never be able to classify objects, but now object classification is a standard technique in machine learning, and Google Picasa can recognize your friends' faces in a group photo. By 1986, an AI could beat the best human players in Scrabble. And of course in 1997, IBM's Deep Blue beat the world chess champion Garry Kasparov. By 2008 AI programs were better than humans at detecting breast cancer from images.

In 2009, we got our first robotic scientist, named Adam. Adam was programmed with our scientific knowledge about yeast and yeast genomics. Adam devised its own hypotheses, and then tested those hypotheses on its own – Adam had robot arms and a fridge full of yeast samples and cameras and stuff like that. Adam assessed the results on its own, and made a novel scientific discovery about yeast genomics without humans intervening at any point in that process. The year 2012 has been no slouch for progress in artificial intelligence either.

Still, many challenges remain. We don't have any machines that can play hockey, or write novels, or do AI research, and some people believe that we'll never get past this next hump and get to superhuman AI. But I think we will. The reason is pretty simple. In short, the human mind does what it does by way of information processing, and machines can do information processing too.

This is the near-universal consensus in cognitive science. Many things look mysterious now, but the universal pattern of history is that things which look mysterious to us now turn out to have lawful explanations and not explanations grounded in some ungraspable magic. Or as Tim Minchin said, "Every mystery ever solved has turned out to be... not magic."

Ray Kurzweil has a great illustration of this in one of his books. A man is frantically scribbling down things that only a human can do and pasting them up on the wall, but then those predictions turn out to be wrong and they fall to the floor. This picture was published in 1999 and since then machines have conquered the "driving cars" thing, and we can see obvious progress in translating continuous speech and cleaning houses.

Basically, if you make these kinds of predictions about things machines can't do, you're going to end up on the wrong side of history. Because intelligent behavior isn't magic, and we'll eventually figure it out.

It looks like we're on a course toward building superhuman AIs, and that is an incredibly big deal. Jann Tallinn, another speaker at this year's summit, put it pretty bluntly, he said, "We became the dominant species on this planet by being the most intelligent species around. This century we are going to cede that crown to machines. After we do that, it will be them steering history, rather than us." And the worry here is that if superhuman AIs are steering the future, they might steer it somewhere that we don't want to go. How do you get superhuman AIs to do what you want?

It turns out this is incredibly, incredibly difficult. We have to invent new math to make that happen – and that, by the way, is the research focus of the Singularity Institute, to work on the most important math problems our species will ever face, problems about how to build superhuman AIs that actually do what we want. By the way, if you know any brilliant young mathematicians, you can tell them, "You can go to Google, you can go into academia, you can go into a hedge fund, or you could spend your career saving the world, and if you want to do that last one, contact" Totally not kidding about that.

The difficulty here, in a nutshell, is that when you're talking about AI, and not a human cognitive system, then there's this huge space of possible designs for a mind. Almost none of them are ones that we would want to be steering the future.

So if you're building a superhuman AI and you pick a mind over there, then it's this giant messy kluge of narrow-AI algorithms and brain-inspired algorithms and it does a bunch of weird things and we can't predict it in advance. You pick a mind somewhere else, and you get an AI that maximizes the number of paperclips in the world, and uses up all our resources to do so.

You pick a mind just a little bit next to that, and you get an AI that maximizes human pleasure. Sounds good, and then the AIs hook you up to a heroin drip and you never do anything cool for the rest of your life.

You pick a mind a little bit further down the line, and the AI creates experiences for us of incredible meaning and value and family and not just pleasure but everything that we care about, except for the one little value we have around novelty. And so it creates for us that one highly-optimized, perfect experience, and then just repeats it for a billion years, and it's just incredibly boring.

You pick a mind just a little bit next to that one, and you get an AI that does everything that we want, everything that we care about, including novelty, but then when the AI rewrites its own code for the 1000th time, that changes its goal function and things go downhill after that and get really bad, because you didn't solve the currently unsolved problem of what's called "reflective decision theory." That's some of the new math that needs to be invented and that almost nobody is working on.

So to build superhuman AI safely, you've got to put in a bunch of extra work, beyond what you'd have to do to build just any old superhuman AI that's got some mind design up there and it's going to do something. 30 years from now, when the U.S.A. and China realize that they could seize a decisive military advantage if they threw a trillion dollars at the problem and get to superhuman AI first, which one of them is going to slow down their progress in order to make sure they get the safety considerations exactly right?

The philosopher David Chalmers actually talked to staff members at the West Point Academy, and they all agreed that the U.S.A at least definitely wouldn't slow down that problem, because a global disaster from American AI is still better than a global disaster from Chinese AI. [laughter]

So this is the sort of problem that we're facing. But if we invest now in solving these open research problems, then when the time comes we might be able to get superhuman AIs that do what we want, and that would be totally awesome.

There's a story I like to tell to illustrate just how awesome things could be if we decide to invest in this research now, and it has to do with the neuroscientist and biologist Robert Sapolsky. In his grad school days, Sapolsky was studying baboons in Kenya. As you may know, baboons are literally the textbook example of a male-dominated, hierarchical, aggressive, violent society. Scientists had lots of reasons to believe that violence was just written into baboon nature, and they had never observed a baboon troop that wasn't violent.

Until the 1980s, when Kenya experienced a tourism boom. Sapolsky was studying his baboons, and there was this tourist lodge built nearby. They dug a hole in the back and threw all their trash in there. Baboons said "Yay, free food," and fought over this pungent bounty every morning. Eventually someone noticed that the baboons weren't looking too good anymore, and what had happened is they had eaten some infected meat and contracted tuberculosis. This kills baboons in a matter of weeks. Their hands rotted away, and they were hobbling around on their elbows. This was just devastating to Sapolsky, and within a couple of weeks half of the males in his troop had died.

This tragic event had an interesting effect though. After that, the troop was basically non-violent. The males would reciprocate when females groom them, and males would even groom other males which never happened. There was a host of the Radiolab show talking about this and they said to a baboonologist this would be like Mike Tyson, in the middle of a fight, just holding back on swinging and approaching Evander Holyfield and just nuzzling him or something. [laughter] It was just inconceivable to baboonologists. But it happened.

Now this was all very interesting, but unfortunately Sapolsky's troop was sort of scientifically ruined by this very unnatural event, so he moved to the other side of this forest and started working with a different troop. But really he was just heartbroken, and he never visited. Until six years later, when he wanted to show his new girlfriend where he had studied his first baboon troop. So he went back there and showed her around, and the baboons were still non-violent, still getting free food from the lodge, the males were still grooming each other.

And then it hit him. Only one of the males that was still in the troop had been through the original event, all the other males were new. All of the other males had come from the dog-eat-dog world of violent, normal baboon-land, and then when they came into this troop, instead of roughing everybody up like they usually did, they had learned "We don't do stuff like that here." They had unlearned their cultural norms and adapted to the new norms of the first baboon pacifists.

Somehow, the violence wasn't an unchanging part of baboon nature. In fact, it changed rather quickly when the right causal factor flipped, and it has stayed changed to this day. Somehow, the violence had been largely circumstantial, it's just that the circumstances had always been the same. Until they weren't.

We still aren't quite sure how much baboon violence to attribute to nature versus nurture or exactly how this happened, but it's worth noting that drastic changes like this can and do happen. Slavery was ubiquitous on Earth for thousands of years, until it was outlawed by every country on Earth. Smallpox decimated populations for millennia, until it was eradicated. Humans had never left the Earth, until we achieved the first manned orbit and the first manned moon landing in a single decade.

Imagine that we've got superhuman AIs, and we decided to invest in making sure that they're going to be safe beforehand, so they're doing what we want. This is a world with hundreds of thousands of better-than-Einstein robotic scientists, in near-perfect coordination, solving all kinds of scientific and technological problems every month.

In a world like that, a lot of causal factors get flipped. In a world like that, humans had never been to Mars but now they make it to Mars. In a world like that cancer had always afflicted our species, but now it doesn't.

It doesn't stop with space exploration and cancer cures though. Consider something as basic to the human condition as pain. Pain is not a law of physics. Pain is due to, one, our current human biology, and two, our ignorance about how to modify or transcend our current human biology. Pain is just a mechanism, stumbled upon blindly by evolution, to inform us that, "You should avoid that." With enough intelligence, we could come up with a mechanism that gives us that information without the pain. That is how robots do it today.

What about death? Death has always been something that has afflicted our species. But is that because death is written into the laws of physics, or is it merely part of our present circumstance? The Turritopsis nutricula, or "immortal jellyfish," is biologically immortal, because it just evolved not to die after a few decades like humans did. We don't die because physics requires us to die. We die because we are not yet smart enough to figure out how to not die. But that doesn't need to be the case anymore if we've got superhuman AIs figuring things out.

Life can be a lot better, and a lot longer than we often suppose. As Eliezer Yudkowsky once put it, "Try convincing a hunter-gatherer from 10,000 years ago that someday people will invent an improved bow and arrow called a 'nuclear missile' [laughter] that vaporizes everything in a 10-mile radius." How ridiculous something sounds to us now is actually not a very good measure of whether it's technologically feasible. And remember, we went from the bow and arrow to nuclear missile with mere human intelligence. We really can't fathom what is possible with superhuman intelligence.

But we don't get all these goodies for free. Almost all the places we could steer the future from here are places that we wouldn't want to go, and almost all the mind designs that we could pick for superhuman AI, even if we're really careful, are mind designs that would steer the future somewhere we don't want to go. To get superhuman AI that does what we want, we have to find a very specific mind design in mind-design space and implement that one. That means inventing a lot of new math that almost nobody is working on right now.

The Singularity Institute actually knows some brilliant mathematicians who can work on these problems and want to work on these problems, and we can't afford to hire them right now. That's the state of funding for the world's most important problem. So yes, I admit it, this talk is a request for your support. If you are already directing your charitable contributions to matters of drastic importance for the fate of the species, then forget I said anything. [laughter] Otherwise, we do need your support, not just your applause. Thank you. [applause]

[Q&A begins]

Man 1: Thank you for the talk. It was very inspiring, and a good summary. I have a question about death. It appeared in the talks for the second time today. I think that death was really important for evolution to have species living a short unit of time so that more advanced and more efficient species survive, and genetically code themselves into their descendents, so I do think that superficial AIs will also need some kind of death to improve in the future, to be better and better. Thank you.

Luke: It's an interesting question. Superhuman AIs are just so different from humans and from other biological creatures that it's not clear to me that death will be as meaningful a concept there. There will still be AIs that can be destroyed and so on, but AIs might be capable in the future of drastically re-writing their own code and becoming something that you wouldn't even recognize as the same AI as before. Also, the future of our world will be in the future less shaped by biological evolution and more and more shaped by technology. So the forces are a little bit changed, but it's a really complicated topic.

[next question]

Man 2: Hi. A billion years ago, some bacteria started splitting and today all of life on Earth is that same bacteria still splitting and reproducing. I feel like that we're still all from that one organism, but that as we come up on the singularity that we're going to create essentially a child race that doesn't really come from that bacteria anymore that's a new species of life that we create. Logically, that child race should either replace us or retire us somewhere in some type of a simulation where we'll be happy forever in a computer somewhere thinking that we're all alive – perhaps it's already happened, I don't know. [laughter]

Do you see the singularity as being sort of a different species of something that we're really crossing a line there from biological evolution into something completely unknown? And how do we make a decision on how to create that since we're going to be in charge of that, what values do we apply? You say that we want to have something that is what we want it to be, but how do we determine what we want it to be?

Luke: You've asked, especially with the last bit there, what is really one of the hardest problems that we have to answer. It's currently unsolved, so this is also part of the research program of the Singularity Institute. It's really hard to reason about the values that we have, especially because we are weird spaghetti-code biological creatures that change our preferences based on whether someone just handed us a hot cup of coffee or something like that. It's really bizarre to try to extract coherent preferences, or what's called a "utility function" from a human. And then even more so, how do we combine the values of different humans and come up with some kind of value system that we would want to give to the AIs that are steering the future? So I don't know how to answer that question. I would pay a lot of money to have the answer to that question, and intend to do so.

[next question]

Woman 1: Hi. A friend from school actually asked me this once, which is, how do you intend to implement this upon all the countries of the world? Supposing we did have an AI, then would you kind of hide it until it was ready, and then release it and have it take over all the governments? [laughter] That seems like a way to make a lot of enemies very quickly.

Luke: [laughs] Yes, this is another strategic question that has to be considered, and it's very hard to think about what the right strategy is right now, because the shape of world government might be very different in 20 years, and we'll also know a lot more about what AI architectures look like, and how quickly they can improve their own intelligence, how quickly they can spread throughout the globe. So this may end up being something that a small team needs to solve, it may require the collaboration of many different groups. I consider that to be an open strategic question that also deserves a lot more thought.

[next question]

Man 3: Luke, quick clarifying question. Is the Singularity Institute's position that no one should be attempting to engineer an artificial general intelligence, an artificial mind, until we have the math for recursive decision processes? If so, how do you support that position from the history of the development of potentially risky technologies? Also, I wonder if you have a comment on whether... this is kind of like the last question, should AGI be developed open source or closed source?

Luke: It would seem wise to me to know how to do something of this magnitude safely before we press the red button and do it, but it seems unlikely to me that we'll be able to get complete global consensus on that. Scientists are not very fond of other people telling them that, "Uh, maybe you should change what you're working on or something like that because there are these ethical consideration."

Another ethical consideration that not many people talk about really is just that, when you're trying to develop an artificial mind that is as complex as the human mind, or maybe even more complex, since we don't know exactly how consciousness works yet, you might actually be creating partially conscious beings that just suffer and experience a bunch of weird situations for a while and then when you turn off the computer you're killing that being. This is another thing that AI scientists might want to consider when they're thinking about artificial minds.

It's a really hard problem. I think that might end up being my answer to every question, unfortunately. [laughter] It's a really hard problem, and we need people working on it!

[next question]

Man 4: Luke, thanks you very much for your talks. I'm a statistician, I work with using social graph data and predicting behavior. I thought that was fascinating but then I heard your stuff and I'm like "Wow, this is cool," so it's rare that I feel like that, thank you. My question for you is more probabilistically, you've raised some great points that it’s really important that we get this right. To a layperson, wouldn't what you just talked about raise more concerns about superhuman AI as a threat to our security, to our safety and to our liberties, and probabilistically isn't the likelihood, conditional that we do develop superhuman AI, that it is friendly kind of low? How would you address that?

Luke: Yeah, unfortunately, because there's so many mind designs that you might pick for a superhuman AI, the default outcome is going to be a mind design that is not optimizing and restructuring the world for things that we care about. That's why it's really important that we do things now that push the probability further in the direction, increase the probability that the first superhuman AI that will be created will be one that will be safe. This involves talking to AI scientists, and persuading them that these are serious concerns, this might mean doing lots of research in human values and how to get consensus of values, this might involve altruism training of some kind. There are all kinds of interventions that might shift the probability so that we are more likely than with no action to achieve a positive rather than a negative singularity.

[next question]

Man 5: Thanks Luke, I thought it was a great talk. I'm going to ask some questions that may be rhetorical, but feel free to answer them however you'd like to. If we build an FAI, a Friendly Artificial Intelligence that has a 99.9% chance of doing what you think it's going to do, do you press the button? If you build an FAI that has a 99.9999% chance of doing what you think it's going to do, do you press the button? What scenarios on the ground, in the world, as far as other AGI programs getting close to achieving their goals, how does that impact the answer to those questions? How does the degree of happiness of the people working on this FAI, how is that taken into all of these scenarios?

Luke: Questions about how certain you can be, of course part of what the Singularity Institute has been advocating for a long time is that we work on what we call "transparent AI architectures," so that you do have a chance to inspect the code and see to some degree whether or not it's going to do what you intend it to do. But judging the probabilities is really difficult. It depends on what you expect the effects to be. You'd have to do a bunch of calculations to try to figure out, given your current knowledge, which option maximizes expected utility or expected value. [laughs] So that's also another open research problem.

The next question that you asked was something about the psychology of researchers working on this, which I actually do find to be a really important question. Humans are brittle and fragile, and if you ask them to think all day about how the world will end if you don't solve this math problem, that might cause psychological problems after 20 years, or something like that. So we actually do think it's very important to develop tools and therapy, something like that to ensure the health of researchers. But this is a good thing for people in general. I have had a long-standing interest in scientific self-help and have written a lot about that subject. Humans are very complex, unfortunately, but I think it is very important, and one thing we consider when thinking about having a productive research team.

[next question]

Woman 2: Computers are already superior to humans in some narrow domains, like doing arithmetic, possibly playing chess, but I think when you talk about superhuman intelligence you're talking about something very qualitatively different than that. So I'm wondering how we're going to recognize it if it ever comes about.

Luke: It's a good question, because of course it won't just be that computes are better than humans at this one particular thing and then we call that superhuman AI or something like that. Computers are going to be much, much better than humans at, you know, 60 skill sets, and they're going to be slightly worse than the best humans at some other skill sets. So there's not a strict thing that's called a "superhuman AI" but we basically think of superhuman AI as AI that is more capable of steering the future than humans are. There may also be a rather quick jump from machines that are just barely superhuman to machines that are very, very superhuman in their intelligence, and therefore much more able to steer the future than we are.

[next question]

Man 6: Hi Luke, I would love to hear your thoughts on how a biologically amplified intelligence would fit into this paradigm. Obviously as we develop new technologies we're going to be amplifying our capabilities with technology, and where does our human intelligence that's amplified by these technologies fit into this? And maybe a little bit of a conversation around a hard takeoff versus a soft takeoff, and the biologically amplified portion of that.

Luke: Of course we're using technology already to increase our intelligence. I've outsourced most of my memory to my iPhone, so I will actually stop people in conversation and say, "Don't tell me, write me an email, because I don't want to use my memory for that." That will continue to be the case with more advanced technologies like brain-computer interfaces, and perhaps we'll discover some new drugs that increase intelligence.

Biological cognitive enhancement is a growing field, an important one, but I think that in the end, anything that's tied to the biological system of the brain is going to fall behind the purely artificial mind architectures, because there's still somewhere in the loop is all this slow neuronal firing, spaghetti-code nonsense that evolution created, that sort of works but is totally non-optimal. So I think that biological cognitive enhancement won't be able to keep up at some point with purely artificial systems.

[end of Q&A]