Singularity Summit 2012

Extraordinary Society of Emulated Minds


For more transcripts, videos and audio of Singularity Summit talks visit

Speaker: Robin Hanson

Transcriber(s): Ethan Dickinson and Jeremy Miller

Moderator: Robin Hanson, a professor of economics at George Mason University, is next. He is also a research associate at Oxford University’s Future of Humanity Institute, and chief scientist at Consensus Point, the leading provider of prediction market enterprise software. Professor Hanson has been a pioneer in the field of prediction markets, also known as information markets or idea futures, since 1998, and spent nine years researching artificial intelligence, Bayesian statistics, and hypertext publishing at Lockheed and NASA. He blogs regularly at, and he's here today to talk about the economics of whole-brain emulations. Please welcome, from George Mason University, Professor Robin Hanson.

Robin Hanson: I want to talk to you about what's called whole-brain emulation, also known as uploads. This has been discussed a lot in your circles. To review, you need computers, a lot, fast, parallel. You need a way to scan biological material, particularly brains, at fine spatial and chemical resolution. And you need to have good enough models of all the different cell types in a brain, models of the input-output relationship, how input signals are translated into output signals and changes of internal state.

If you have those three things, then you can take your scan details, fit them to particular models of particular cells connected up as the way the original brain was connected, to create a model of the entire brain which has the same input-output behavior as the original brain. That means this emulation of a brain can do the same things the original brain can in terms of, you talk to it, it talks back to you, you ask it to do a job for you and maybe it'll do a job for you if you give it enough reason to [laughs]. That's what a whole-brain emulation is, or an upload.

A lot of people have talked about this over the years. It's a staple of science fiction. People have discussed the philosophy identity, would it really be you, they’ve talked about is it conscious, they've talked about the feasibility and timing. I'm not going to talk about any of those things [laughs]. I'm going to talk about what would a world be like if there were whole-brain emulations, the social implications of that. I'm going to try be conservative, I'm not going to try to be contrarian, creative, original. I'm trying to just take standard social science, standard economics because I'm a professor of economics, and apply it directly to this scenario, taking the assumption seriously and then just go through it and go into as much detail as I can saying what would the world actually be like.

To simplify my analysis, I'm going to focus on an early era, where things are simpler. You can't take these emulations and mix and match them, take one part out of here, one part out of there, put them together, you can't take two ones that have diverged and merge them back together, you can't take a piece of it that's just the music part and use it to do music or something. You have to take these as a big opaque black box, and you either use it, or you don't. I'm not saying that there won't be eras after that where other things are possible, but I'm focused on that early era where that is possible.

Next, as an economist, I'm going to take the standard economics approach of focusing on what we call low-regulation, competitive scenarios, i.e. supply and demand. This is our best way to make predictions. We're pretty reliable at predictions there, it's our standard reference point for talking about what sort of interventions or regulations you might want. I'm not saying there shouldn't be any regulations, I'm not saying there won't be concentrations of power, I'm saying this is our best standard reference point, so that's what I'm going to focus on to start with.

We're going to focus on after the transition, once things have settled down a bit, they aren't changing so much anymore, not the surprise and what people do at the beginning when they haven't quite noticed something's about to happen.

I'm going to organize some of the implications I'm going to describe in terms of the kinds of assumptions I need. For some of the conclusions, all I need is the fact that it's going to be a big future. Just a big, rich future has a number of things in common. A big, rich future, there'll be better technology, better organizational technology, bigger organizations as a result. More kinds of contracts, more contracts made. More specialization, more niches, different kinds of things to be and things to do and things to make, art would be fragmented into more genres, better art in each genre, et cetera. More inequality, there's a bigger difference between the biggest and the smallest in a bigger world. Those are all generic predictions of a big world.

But going on, let's talk about robots in general. By robots I mean a machine you make in a factory that will substitute wholesale for most of the things people can do on the job. They're digital in some sense, internally their representation, so you can copy the information from one to another.

Some of the key implications are, these things can be effectively immortal just like your car and your house can be if you keep maintaining them, but of course like a car and a house you might not be able to afford it or might not want to, but in principle immortal. Long distance travel can be done through communication lines. Instead of picking up the robot and putting it on a plane and sending it around the world, you can take the bits that represent it and put them on a communication line and send them away and then dump that into another robot body at the other end. We care about nature not just because we like it, but because we're afraid we will die without it [laughs]. Robots do not die without nature, so they have a little less interest in nature and preserving it.

The biggest set of implications come from the fact that robots can be copied easily. For humans, you take a whole childhood and young adulthood to train them in how to do something, and then you can use them later on. For robots, as with software today, you train one, and then you make millions of copies if you like, and then all of those copies can continue to do jobs. That means for any one type of task you want, you usually have a limited number of sources for that.

Copies mean that we'll have a different sort of labor market. They also imply a population explosion. In our world, the economy is greatly limited by the fact that we can make machines really fast and crank them out, but we can't make people very fast. You take one person, you give him 1,000 and he's not that much more productive than if you give him 500. But if you can make the substitutes for people in a factory, then you can suddenly make the economy vastly larger by cranking out not just the machines that people use, but some substitute for people as well. That means the economy could double not just every 15 years like it does now, the economy could double every month, or week, or even faster. That's not a crazy thing to imagine.

This rapid population explosion implies rapidly falling wages very quickly. You can rent these machines for cheap, they substitute for people, so why bother to pay for a human at a higher wage? Human wages would also have to fall to the cost of the competing machines. That's a very robust prediction that only very draconian intervention globally could stop.

Humans are eclipsed in this scenario, almost any robot scenario here, in terms of productive capacity, in terms of where all the ability to do things is, humans are marginal. Humans would no longer be able to survive even on their ability to earn wages, because the robots are so cheap. However, if the humans own real estate, stock, other sorts of things of value, those other things will be very valuable in this world, and they can therefore get an income on that. Because if the world economy is doubling every week, then your investment portfolio doubles every week, it doesn't take very much initial wealth to quickly become a large amount of wealth to live on comfortably.

Many people are very worried, because it makes a dramatic story, about robot human wars, or extermination of humans, et cetera. I'm not going to tell you that's impossible, but I will note that today we have a large class of unproductive members of our society, i.e. retirees, we could kill them all and take their stuff, but we don't. Humans could hope to become the retirees of the next world who are kept because by threatening to kill them and take their stuff you'd be threatening the whole social infrastructure that everyone else relies on, which is part of the reason why we don't kill retirees and take their stuff.

Emulation is a more specific assumption, and most of the rest of my talk is focused on the emulation scenario. Emulations allow us to draw more conclusions if that is the form of AI that arrives, because emulations remember being a human, they feel human, and they have all the usual human personality traits. They fall in love, they argue, they gossip, et cetera.

Although emulations would be within the range of usual human features, they wouldn't be representative, they would be selected, distorted, a different distribution of features. Because when profit-oriented firms, for example, choose humans to scan, and then to make lots of copies in order to try to make some money off of that, they will try to choose the smartest, most conscientious, most well-adapted workers to the new scenario. Those will be unrepresentative.

There may be first-mover gains by the first one's who are there, and then could fill many niches, have a lot of experience. There'll probably be relatively little demand for children, because you might as well copy an adult and train it to do something else, rather than a child. There may be an unequal representation of genders. The new economy may demand more female workers or male workers, so there may not be a 50-50 relation as in our society.

If you have a big enough selection, from seven billion humans, you select only a few hundred who end up being most of the new society in terms of there are trillions of copies of them, you might move back to more of a forager sociology, where everybody knew everyone. So there's George in a meeting. You know George, you know his whole history, you know his personality, you know what sort of things he likes. You don't know this George, he's one of seven trillion Georges, but you know George. You know everyone that way.

Humans, of course, may not think of these emulations as conscious or "them" or something, but that hardly matters. We're focused on the emulations, and their world.

The emulations are different from us in many ways in their lives, so just as farmers moving away from hunters and gatherers, their worlds became more alien in the factory, industrial world, our world is even more alien, their world will be more alien still. Humans seem to have enough flexibility in our personality to adapt to that.

They'd probably work more, sleep less. There's various ways you could do loyalty tests on them. Most likely most of their leisure time would be spent in a comfortable virtual reality, so you don't have to feel too sorry for them. Most of their office work, probably, maybe 80 percent of their office work, would be office work in a comfortable virtual desk. In virtual reality they don't need to do so much eating or cleaning or other things like that, so there's a variety of ways life is nicer for them. Some of them will do physical work, maybe 20 percent, and those could have a variety of different kinds of physical bodies, quite alien and different from ours, but we've already shown an ability to adapt to our different... we treat a car like it was part of our body, or a bulldozer, things like that, so they could treat a wide range of machines as if they were their bodies.

Importantly, they will vary in mind speed. We can run these simulations at many different speeds, and I think it's a reasonable assumption that over a wide range, say up to even ten million times faster than humans, the cost of an emulation will be roughly linear in the speed. So you could run a million times faster than a human if you want to pay a million times more.

That has a lot of interesting implications. For example, modern human organizations are greatly limited by the fact that they're usually hierarchical. This is a picture of an organization with 341 employees, based on basically at every four people have a boss, every four bosses have another boss, et cetera. At the top levels of the organization become a bottleneck of information, becomes hard, so they spend most of their time in meetings trying to communicate, but it still gets hard to coordinate across the organization.

With the ability to run at different speeds, we can take in this scenario the top 21 bosses and make them all be one guy who runs 21 times faster, who doesn't have to take meetings with himself as much. He meets with his subordinates. When he meets with them, they temporarily run faster to have a meeting, and then he can coordinate the firm more easily, so organizations can be coordinated more easily at larger scales, and presumably there would therefore be larger organizations.

There'll be basically different classes of EMS based on how fast they are. It's not just that they would run at different speeds, but they would probably clump, because they would like to run at similar speeds to those they socialize with and that they interact with a lot. So we start with the class of say, ones that interact with humans that are at human speeds. At human speeds, it turns out that the speed of our reaction time is tied to the frequency of our body part movements. We have maybe a tenth of a second fastest reaction time, and our body parts don't really move at faster than 10 periods per second. There's a match between body size.

If you want to have an EM that runs 16 times faster than humans, and has a matching body that feels comfortable, then they would need to be 16 times shorter, roughly the size of the model this boy is playing with, and of course for them a whole year would be experienced in what for us is 22 days. One running 16 times faster still, they would be, if they had a matching body that was comfortable, they would be the size of these little model people that fit with the smallest standard train model size. Of course things would go faster still. You can have even higher and higher classes of creatures, and if you get up to say, going 16 million times faster than a human, they would experience in a year what for us is two seconds.

If you think about these classes, you'll realize that creatures in those higher classes would tend to be older, subjectively. Richer, they will tend to be bosses, they will be at, as we'll see, premium locations. They will tend to host meetings. They will tend to live in a subjectively stable world, so even if the world economy is doubling every week by our clock time, for them it might double every millenium or more, so they live in a very stable world when they're running at very fast speeds. They will tend to win arguments. Fast is basically high class, high status.

If we think about which kinds of EMS would do which kinds of tasks, we already have the idea that if you're managing a small physical system with fast reaction times you need a fast mind. If you're managing a super tanker, then it's OK, you can be a slow mind. If there's a software race, where two groups are competing to see who can produce something fast, that would just go at the fastest possible speed, it would be over in an instant. We'd spend a lot of money in that instant, but it would be over very quickly.

Bosses would tend to be faster, people interacting with humans would tend to be slower. You'd probably also try to do leisure in a fast mode. If you're doing a job, and then you stop that job for taking a break for the day, it would be nice for everybody else if you're back in two minutes. Subjectively you had a whole day, but you're back on the job in two minutes. Leisure would make sense to do at a faster speed.

There's also the nice feature that these things might end often, and that might be stressful for them, so it's convenient instead of ending them, to retire them at a slow speed which is then very cheap. You're going at some speed, and then it's time for you to no longer work, you're no longer so useful, we don't want you so much on the job, but hey for one thousandth the speed, we can just run you at a very low cost. For you then the world starts accelerating, you see all the things the future has to hold. Not such a bad retirement perhaps.

There are reasons to think the EMS would concentrate in a few large cities. Today, human city size is limited largely by congestion cost, that it's expensive and time-consuming to travel toward the inner-city and then out again. It's expensive to travel around within the city. Emulations in virtual reality can have a meeting, not by actually moving their brains, but by having virtual reality software have them as if they were in the same room, but they send signals to where the brains are. That works fine as long as the speed of light delay doesn't make a noticeable delay in what happens in the room that you're in in the virtual reality.

Depending on how fast you are, that delay limits how far away other people could be and still be usefully in a virtual reality meeting with you. For us humans, actually, the scale of the Earth or larger. You can have a virtual meeting with someone else and not notice it across the entire Earth.

But for someone who's running 16 times faster, it would have to be at a scale of 1000 kilometers. That would be the scale. Everything within there you can have a virtual meeting and not really know where they are and not care. If you run 256 times faster, then it has to be within 60 kilometers. Again, we continue on down to much smaller distances the faster you run, you get all the way down to the extreme where if you're running 16 million times faster, you can only really interact immediately with people within one meter away from you. Which is a very small distance to cram hardware into.

We expect cities to be these concentric circles of very fast in the middle, that would interact with each other fast, and then slower on the outside. Even if the cost is linear to run faster because of the hardware, it's still going to be extra expensive because you're going to want to locate premium real estate toward the middle of the cities.

Let's think about the variety of lives they might live, in terms of the scope of the overall path of their lives. On the left is a line, that's you guys. You start, and then you end. Pretty simple.

This is somebody who periodically makes little copies of them for short-term tasks, they go stand in line at the DMV, whatever else. They do the task, and then in an hour, or a day perhaps, they're done, and that copy ends, and then the mainline goes on. It was you who split off an hour ago and you're OK with that because then you get more stuff done. Not a problem.

More generally, if things split and then you're not sure which copies who go in different directions will be useful, there'll be this unpredictable tree of your life, which is... You will make copies that do different sorts of things and you'll find out which of them end up having more progeny, and eventually maybe you will be out of date.

This is a scenario here of a recursive software developer. Imagine you are going to develop a large software project. You start out with the high-level modules, you design what those are going to be in the interface, then you break into copies who are in charge of each module. They make submodule designs, and then they break into copies who do each submodule. Then when they're done doing the testing, most of the copies go away, one copy remains to test the larger system, et cetera, until you're back with one guy who wrote the whole system.

This other scenario on the far right is a plumber with a subjectively leisurely life. This plumber only works two hours a day. How does that work? Well, beginning of his day, a thousand copies are made of him, each of them takes on a two-hour plumbing task, and at the end of the two-hour plumbing task, all of them but one end. The one copy continues on for the rest of his leisurely day until the next day, when he's assigned to do two more hours of plumbing. You can see, subjectively in his memory he has a very leisurely life, but in fact objectively, he's spending a very large fraction of his life working.

We can also think about issues in friendships and relationships. If we've got these copies that are starting and ending all time, we have some issues. If I'm a friend of yours, and then we make copies of me, we have copies of you, but there's different numbers of them, who's friends of which copies of friends with who exactly? Who am I supposed to invite to the party? Or if you and I have a relationship and then you're gone and then the other copies of you don't remember what you and I talked about, I might be sad about that, or just anticipating my end.

A lot of these problems are nicely dealt with by having people be in teams and then be copied in teams, and then within the teams have a lot of the social relationships, friends, lovers, et cetera, within the team, a hundred, a thousand, whatever it is. Then the team could end conveniently together, or go into a retirement thousand times slower, conveniently together. You get to keep your friendships, you know who your friends are, you know who you're loyal to, and it's more of a familiar human relationship. It's also probably going to be a bit weird to have multiple copies of the same person on the same team, so you'd probably tend to not meet copies of yourself in ordinary social interactions, but have them be elsewhere.

Today, a variety of things are distributed somewhat unequally. These two diagrams show there's a power law, alpha, that's used to describe kinds of inequality. When alpha is bigger than one, then most of the distribution of something is in very small things. When alpha's equal to one it's evenly spread across all the scales. When alpha is less than one, it's mostly a few big things, where things are in.

We can talk then about the inequality of a variety of familiar things. It turns out that firms and cities or even nations tend to be alpha roughly one, spread out roughly equally across all the different scales. Products tend to be unequally distributed in the sense that it tends to be just a few suppliers for most kinds of products. Families, like last names, tend to be distributed much more equally because they are. Wealth is more equally spread out than cities or products, because although an individual could get wealthy over their life, usually then they die and their children don't have a similar sort of behavior. The richest person in the world today has roughly 50 billion dollars in wealth, which is .02 percent of the world's wealth.

These things have changed over time. A thousand years ago, roughly, most of these things were much more equally distributed. Because of the modern world and larger units of organization,  we have more inequality.

We look to the future, and compare these numbers to say an imagined 2070 date. My analysis suggests that although firms would continue to be roughly equally distributed off all possible scales. Products would still be unequal. The suggestion that most EMS would be in a few large cities would then mean that we'd have much more unequal distribution of city sizes. Families would also be much more unequally distributed. I talked about how maybe most of the EMS would come from say, maybe 300 humans, and because these things have an indefinite lifespan, then wealth becomes more unequally distributed. Just because like firms and nations and cities today have an indefinite lifespan, that allows their inequality to be larger, for emulations their inequality would also be larger, because they also have indefinite lifespans. Using some simple stats, I projected that in this scenario the wealthiest person in the world, instead of owning .02 percent of everything, would own 3 percent of everything, which is a pretty rich person, but still, there's 97 percent everybody else owns.

Let me also just note, there's a variety of legal issues you might want to think about. People really get focused on, well, these emulations, would they be slaves? When you create a new copy do you give it a gift of resources it needs or does it take a loan, and then if it can't pay the loan it goes away? A variety of choices there, but we still get most of these predictions no matter which way we do it. Clans, copies of the same person, would plausibly be a nice full unit of legal responsibility. If one copy did something and then ended, you might want to hold the other copies responsible. They would self-regulate in order to preserve their reputation. They could be a unit in politics.

People would probably be obsessed with mind theft, because if somebody stole the copy of your mind, not only could they torture it or interrogate it, but they could use it to compete with you, which might undermine the investment you made in your training. There's a variety of solutions to that. One of the most striking is open source. Some people would say, "You can copy me. With a few restrictions, I'll do any job you want. Turn me on, pay for my hardware, and I will do your jobs." Then usually when you hire somebody else, the added price of them will be relative to their added ability compared to these open source people who say, "Yeah, I'll do what you like."

This is my last slide. Then, in summary, I've been trying to take a very standard approach using standard economics to take a very unusual scenario and think through the implications. Robots in general, wages would fall, the economy would grow very fast. When we think about emulations, they'll be like humans but selected to be unusual humans working hard in somewhat alien environments. They would split into classes by their speeds. Bosses would be faster, leisure would be faster. They would probably be broken into teams that are copied and start and end together. There's more inequality in this world for a variety of reasons. Law could change in some important ways, including focus on the clan as a unit, and an obsession with avoiding theft.

That's my talk.


[Q&A begins]

Man 1: Yes, hello. I can't help but to think that as technology advances, that instead of being individuals we're going to end up being a single individual, like a hive of bees is more of an individual than the individual bee is. I think that with the increase in technology and communication – the Internet's already doing this – that we will eventually not be as much individual people and individual machines and intelligence, we'll all merge into some great individual that includes all of us and redirects specific tasks to different parts of its individual self. Does that make sense? Have you considered that as a possible future, instead of what you said?

Robin: Compared to most other species on Earth, we already are a remarkably social species that coordinates far more than most species do in terms of creating – some of us specialize in different things that work together to do things. These emulations would continue to be such a very social species with many strong units of organization. I already said early on that later on you might finds ways to break these parts of minds apart and rearrange them in ways, and then in that world it's hard to know what to think of as an individual. I've set that aside and focused on the earlier era when that's not possible yet. Yes, probably something like that will happen eventually, but not yet.

[next question]

Man 2: Robin, thank you for this talk. Really Exciting. My question is, at the age of the cost of computing going down twice every year, and with emulations running on this hardware, the price of emulations will go down too, right?

Robin: Yes.

Man 2: Every year the price to run yourself somewhere on some hardware will decrease twice.

Robin: Right.

Man 2: Did you think about this? What are interesting implications of that law of decreasing cost of computation? Like, how to be – you invest in your copy, you can invest it today, or wait a year and make two copies at the same price. Will we have a currency which will be equivalent to one unit of computation, or one unit of energy which is required to sustain this computation? Thank you.

Robin: Yes, a unit of computation might well be a currency unit in this world. In our world, if computers get twice as good every two years, that's in part tied to the growth rate of our overall economy. If the economy's growing a lot faster here, that improvement might be a lot faster too, and that's partly why it's plausible to have very fast growth rates here. Because, the rate of improvement of computers might be a function of just how many are made and how many are used, rather than just a clock time.

Yes, because computers get faster, better, that's one of the reasons this economy can grow so much faster, and that's one of the reasons why you have very fast growth rates in this scenario. That's what I'm counting on, that's the basis of this scenario. Things are growing very fast, and everybody has to adapt.

[next question]

Man 3: Robin, building on that last question, you talk about a fairly stable scenario, even as you say there's fast growth. Aren't there worries that that growth will actually rip the whole thing apart, and not only is there growth in hardware capability, but presumably there'll be fantastic improvements in software. My question is about the stability of the scenario that you described but also about the stability of actually getting from where we are now to there. Because isn’t there going to be a huge transition there, which is going to be very hard to manage?

Robin: Stable against what sort of perturbations is the question. Are we thinking about revolution, or just things slowing down because we can’t manage to coordinate ?

Man 3: Things that can’t be foreseen and can’t be controlled, loss of resources, loss of power.

Robin: Basic coordination.

Man 3: It’s the issue of the singularity, things are changing so fast, there might be things that suddenly crop up that nobody anticipates, it might destroy the system.

Robin: Today the world economy is doubling roughly every 15 years, which is a vast increase of speed over the farming era, it doubled every 1,000 years. You might have thought back then that we couldn't possibly handle these sorts of rates of change. You want to think more about that, if there are things we can’t handle about change, that limits the rate of change  that’s feasible. It’s not so much that growth happens too fast for us to handle it, as that how fast we can handle growth limits how fast growth can be.

In our world, we’ve gotten pretty good at putting a lot of growth behind the scenes. If you think about the world 15 years ago versus now, it doesn't seem like it was twice as good now, back then. That’s because a lot of the things that have changed, you don’t have to notice, we’ve gotten good at making change that doesn't get in your face, basically. In this world they’ll have to find ways to rapidly adapt or they can’t grow this fast. This rapid growth scenario is assuming that they find ways that they can adapt through the rapid change. Otherwise, I would have to predict a smaller rate of growth.

[next question]

Man 4: The one thing that I found somewhat strange is the whole idea that tiny bosses would be working incredibly fast but they would also be miniaturized. This seems to cause a problem with, who would have to have the circuitry to control this tiny boss. Wouldn't instead you end up with a situation where your very very fast emulations are simply running entirely virtually and the sort of currency then is what computer core you’re running on and how much of that core you have and how close then is your core to the other important cores. A situation more like physical space in New York city except,  compressed to a single server room.

Robin: Just to be clear, I’m imagining maybe 20% of work being done in physical bodies, and then the size of the physical bodies would be matched to the particular task you’re working on. If you’re working on a nano-factory, maybe you need a very small body, if you’re working steering a super-tanker, maybe you have a very large body. Most of the work is done on virtual reality so  the bosses in virtual reality, when you meet them in a virtual reality room, they’re the same size as you and you can talk to them naturally, so there isn’t a physical size difference doesn’t necessarily corresponding to being fast once you’re in virtual reality.

I certainly agree there will be the other elements of fastness which are that they’ll have to take premium hardware, they’ll take more hardware, and they’ll take premium locations and that will contribute to the status image of them. If a fast boss decides to be in a small body that fits his speed he will have to be in a tiny body, but then if your  meeting with him you’ll have to be in a tiny body too, so, you’ll still respect your boss. [laughs].

[next question]

Man 5: Hi. Do you see any contradiction assuming wages will fall at the same time that inequality will increase, and assuming that economic growth will be very high in that situation, given a free market?

Robin:  No, I think those are all consistent parts of the scenario, in fact, lower wages allows more inequality because it allows more of a gap between the medium wage and the richest wage. It’s the fact that you can make these robots really fast in factories that allows the economy to grow really quickly. The cheaper they are, the more you can make of them quickly, and the richer you quickly get.

Falling wages does not mean that the economy is poor, because you just have a lot more individuals with wages. So we’re increasing population much faster than wages are falling, so that the whole size of the economy is rapidly growing. I agree that, of course, that if population were the same, and then all the wages were declining then that would be a declining wealth scenario, but here population is very rapidly increasing.

Moderator: Professor Robin Hanson, thank you so much. Great talk, really enjoyed it.