Singularity Summit 2011

Balancing the Trichotomy: Individual vs. Society vs. Universe

[Video] [Audio]

For more transcripts, videos and audio of Singularity Summit talks visit

Speaker: Jaan Tallinn

Transcriber(s): Matt Cudmore, Ethan Dickinson

The MC: Jaan Tallinn is here. He describes himself as a hacker, entrepreneur, investor, and physicist, in that order. He grew up in Communist Estonia, but his entrepreneurship has repeatedly landed his small home country — population 1.35 million — in the international news. In 2000 he helped create the infamous file-sharing platform Kazaa, which according to some sources accounted for more than half of all Internet traffic at its peak. In 2003, he created Skype, a free online calling system that has since grown to account for 13 percent of all international calling minutes. Jaan is still a lead system architect at Skype, and he is considered one of the world’s foremost experts on P2P technologies. He is also a partner and cofounder at the seed-stage venture firm Ambient Sound Investments, a member of the Lifeboat Foundation advisory board, and a member of the Estonian president’s academic advisory board. Today Jaan will discuss the huge consequences that our present-day decisions will have for future generations. Please welcome Jaan Tallinn. [applause]

Jaan Tallinn: Good afternoon. The curse of speaking near the end of a two-day conference of like-minded people is that you’re bound to repeat quite a lot of things that have been said. However, having myself developed a spaced-repetition software to teach me things, I can only appreciate the opportunity because I know that only by repeating things, people will really remember the important points. Without further ado...

[resolving tech troubles]

My presentation has four parts. First, I will introduce what I call “Level 3 challenges”. That is a new class of problems and opportunities that will likely shape this century, and possibly the rest of history. Second, I’m going to show why people are ill-equipped when it comes to dealing with them. Third, I will look back on my life, and show how I personally discovered those challenges. And in the final part of my presentation, I will describe a force that could stand up to those challenges, and basically describe the start of a movement that I seem to be witnessing.

First, let me explain what I mean by “Level 3 challenges”, challenges that increasingly powerful technologies create, and that baffle our intuitions. To do this, let’s go back to the year 1983. On September 1st, Korean Air Lines flight 007 — yes, ironically named — flew into Soviet airspace and was shot down. Among the 269 casualties was a sitting member of the U.S. Congress. What happened next is considered one of the most volatile moments of the Cold War. Tensions rose to an all-time high, and the Soviet Union started preparing for war.

Three and a half weeks later, the Soviet early warning system picked up several incoming ballistic missiles. The officer in charge, according to the protocol, he was supposed to react immediately by launching a full-scale counterattack. He did not do that. The alert, of course, turned out to be a false alarm. Needless to say, that officer in charge decided the fate of the world that day. I would not be speaking here had he followed the protocol. I would probably not even exist, and quite possibly neither would you.

His name was, and still is, Stanislav Petrov. While he has received some recognition for his act, for literally saving the world, the reward he has gotten is comparable to what one would get for saving the life of one person, which is crazy.

Two important lessons can be drawn from this example. Contrary to our intuition that society is more powerful than any individual or group, it was not society that wrote history on that day 28 years ago, it was Petrov. The second lesson is that while our intuition says that society is wise and can deal with any challenges that are thrown at it, the society didn’t even notice that something was thrown at it that day. It still hasn’t.

It is easy to see, though, why we have such intuitions. They date back to the day when our most advanced technology was a stone attached to a stick, and the life of every generation was pretty much a copy of the one that came before it. Recipes for major challenges were firmly embedded in cultural memory. We knew that society would be able to deal with problems, and we were right.

However, if you consider upcoming technologies, that are way more powerful than the one that Petrov was in charge of, those intuitions clearly no longer hold. Instead, our future is increasingly determined by individuals and small groups wielding powerful technology, and society is quite incompetent when it comes to predicting and handling the consequences. The cultural memory is completely blank when it comes to them.

One way of looking at this is that by catching up to the timescale of human generations, the rate that technology can change has driven a wedge between current society and future society by making them different. Instead of the classic dichotomy of individual needs, such as health and family, versus the interests of society, such as law enforcement and wealth distribution, we now have a trichotomy to consider. Individuals, versus society, versus future society.

This way we get a three-level hierarchy of challenges. On the first level we have individual challenges. Second, we have traditional social challenges. And third, challenges that our society lacks a cultural memory for yet, which clearly affect the interests of future society. These are what I mean when I say “Level 3 challenges”.

Because we are social animals, evolution has had over 100,000 years to make us pull our weight in the individual-versus-society dichotomy. However, the instincts we have picked up along the way can actually actively resist our ability to contemplate, much less deal with, Level 3 challenges.

To further illustrate that, I’m going to use a domain that I’ve been exposed to in the last six years or so, that is the domain of very successful people.

The point of becoming economically independent is like an end of the railroad. Life has been in the service of a particular company working towards concrete goals, so that it becomes largely unconstrained. Therefore the actions that people take at that point can actually read a lot about their true nature.

Interestingly, the same three-level taxonomy of challenges I’m presenting here can also be used to categorize such people.

On the first level, we have people who start pursuing personal interests, that is mostly trying to optimize their own wellbeing and happiness. Interestingly they often identify their happiness with the rails that they used to be on, so they simply continue along the imaginary ones. I’m talking about serial entrepreneurs and investors here, people who become attached to working hard and making money.

On the next level, we have people who become socially conscious, and get involved in politics, non-profits, or philanthropy.

Importantly, I think that both Level 1 and Level 2 activities can be quite commendable, and they are in no way in conflict with each other or with Level 3. However, in optimizing the balance between Level 1 and Level 2, evolution has played a nasty trick on us. A trick that prevents the vast majority of people from reaching Level 3 where they could actually deliberately address long-term issues. That trick is drawing people into a social status reinforcement cycle.

Because the individual-versus-society dichotomy has been around even longer than our species has, we have evolved reward mechanisms for doing things that society considers good. We get awarded the universal currency of social status as a result. And as with any sufficiently powerful reward mechanism, people get addicted to it.

As I see, from the perspective of having an impact on future society, people stuck in the social status reinforcement cycle are bogged down in at least three ways.

First, they focus on short-term projects only, because they want their social status fixes to come in as frequently as possible.

Second, since social status reinforcement is not entirely proportional to your achievements, people in the social status reinforcement cycle have reason to be less ambitious. For example, in terms of social status reward, there isn’t much difference between Elon Musk’s attempt to get our species off this planet, and building a local animal shelter.

Third, the amount of social status reward depends on the perception of other people, so your activity has to be easy to understand to the public at large. I see the Gates Foundation as an example of an organization that tries to be as efficient as possible while limiting themselves to the domain of issues that everybody on this planet can easily understand.

Coming back to the types of people, like I said I’m not trying to belittle what Level 1 and Level 2 people are doing. Addressing Level 1 challenges can be clearly necessary for personal sanity and effectiveness. Also, dealing with Level 2 issues can earn you the social status that then you can apply towards your other goals just like money basically.

I’m just saying that future societies will look back on us and feel depressed because of the actions that we did not do. Luckily there are Level 3 people on this planet and the ranks are growing. There are people who truly try to empathize with the needs of future societies, and consider those interests when they plan their actions.

I haven’t yet figured out what it takes for a Level 2 person to break out of the social status reinforcement cycle, and graduate to Level 3, but I can tell you what happened to me. Time for some personal reflections. Here is my life story in a nutshell. [image of nutshell on screen; laughter]

I was born behind the Iron Curtain to a nation that was on the verge of dying. I had no illusions about having a global impact on the world. After all, the Soviet Union was a country that you needed a visa to get out of. [laughter] Surprisingly though, just as I finished high-school, I'm still wondering about this coincidence, it’s weird. As I got ready for independent life, the empire crumbled.

I found myself developing computer games for an international market with friends. We founded a small computer games company that pretty much constituted the entire Estonian computer games industry. [laughter]

As our games business petered out though, because of their computers getting more and more powerful, and the games changed in their nature, we ended up in Internet programming, and part of the team went on to create the file-sharing software Kazaa. This was my first taste of global impact. I still remember sitting in my modest apartment in a nondescript Soviet-style apartment block with the Kazaa team. We were sipping wine and thinking that “Oh, so this is how it feels to be responsible for the majority of Internet traffic out there.” [laughter] [applause]

And of course, the same team went on to create Skype. When people ask me, “How did it feel to be part of something as big as Skype?” I reply, “Well, familiar.” [laughter] What was not familiar though, was the moment in 2005 when Skype was sold to eBay, and I saw the track coming to an end. A friend even asked me, “So what are you going to do now? Having completed your life’s work at such an early age.” [laughter] Even though I laughed it off then, it got me thinking, indeed, what could I possibly do that would be bigger than Skype? Indeed what does the word “big” even mean in this context?

Luckily, after a few years, a couple years ago, I stumbled across the Overcoming Bias and Less Wrong blogs. I learned how our biased instincts are running the world, and why it is important to strive towards rationality.

Also it became clear to me that in order to maximize your impact in the world, you should behave as a prudent investor, you should look for underappreciated areas with huge potential. And the biggest pool, indeed an ocean of underappreciated causality, I found in the so-called existential risks domain. Things that might go horribly wrong, and wipe out our entire species, which of course I presume is the ultimate Level 3 challenge, because I think the first thing that every future society cares about is not being dead. [laughter]

Furthermore, I was particularly drawn to AGI problem studies — problems in the field of artificial general intelligence — since that is the only subdomain of existential risks where the stakes are high both ways. If things go well, they go really well. Indeed, we might solve the other remaining Level 3 challenges that way. Also, since this domain is notoriously hard to reason about, it is pretty much guaranteed to remain underappreciated. For example, the majority of the people in the AGI field seem to be focussing on the problem of how to build one, which as problems go is a tough one, but it’s peanuts compared to the real problem we face, how to make an AGI do what we want. So that’s where my focus is today.

In particular I identify three channels I could use to have an impact in this domain. First, donate money, so people who work on the problem could do so. Therefore I’m one of the largest donors to the Singularity Institute now. Second, follow the ongoing research and proposed solutions so far. As such I’ve been talking to both Singularity Institute and Future of Humanity Institute in Oxford, as well as the AGI developers directly. Third, capitalize on my street cred, get people to listen and make them aware of the big AGI problem as well as Level 3 challenges in general. Hence my presentations and networking.

I’m happy to report that the problems do seem to resonate with people, and quite often get the question, “So, what can we do?” So I’ve been thinking how to scale things up. Luckily, I’m not the only one thinking about this. In my travels over the last year, I met a significant number of people and groups that are very concerned with Level 3 issues, and are going to do something about it. The wheels are turning, so to speak.

One trivial yet critical piece that seems to be missing, though, is a way for those people to identify themselves against the background of Level 1 and Level 2 people. For example, here’s how I struggle when I try to identify myself. [profile and text on screen] While I think that “singularitarian” is close to what I have in mind, it is much too ambiguous and has really strong sci-fi connotations that can be very counterproductive. Similarly, although I consider myself part of the rationalist movement, I think the term “rationalist” is still too broad for this particular purpose. I’m lacking a good label. I’ve seen Level 3 people start using the term “sane” [laughter] both as an inside joke and a reflection of the sad truth. Obviously such a label would not go down well with other people. [laughter]

So I would propose calling the movement “CL3 Generation”, because it would reflect the fact that in order to be part of that, you would need not only to be aware of the Level 3 issues, you would also have to be aware of the inadequacy of traditional solutions that previous generations have used. Appropriately, there seems to be a significant correlation between the age of people and their ability to understand Level 3 issues. I’m saying just correlation, not an absolute. So the members of the movement are more often than not productive young people.

In addition to giving you the warm and fuzzy feeling of having identified yourself with a group, I see that the label can help the movement in three other ways.

First, fight dilution. If you belong to the CL3 Generation today, and unless you are part of an organization such as Singularity Institute, you are likely surrounded by peers who don’t know what the Level 3 issues are, or just don’t care. Having a movement to identify yourself with can help really alleviate that.

Another way that such a label could be useful is to declare intentions, and therefore paving the way for better cooperation because you’re more predictable. For example, I’m pretty predictable, because I care about Level 1 and Level 3 much more than I care about Level 2. For example, when I was approached a few weeks ago by a museum in Estonia, they offered me a board seat, and I predictably declined because while it would have been a respectable challenge it would have been clearly a Level 2 challenge.

Last but not least, it would be invaluable if such a label would evolve into a positive meme, so that even people who don’t understand the Level 3 issues would recognize that this is something serious and worthwhile, not unlike the global warming meme has developed today.

Furthermore, similar to the causal channels I listed earlier, I can see different roles that CL3 Generation members could take.

Donors. Donors are people, charities, and commercial organizations that support the other members of the movement. It doesn’t even have to be in a monetary form. For example, I consider the startup Quixey as a CL3 donor because it functions as an employer and a hub for CL3 people.

Similarly, Michael Vassar, Rob Brazell, and myself, as well as some other people in this room, are in the process of starting another startup that would hopefully graduate into a CL3 donor.

CL3 researchers obviously would be people doing direct research into Level 3 issues.

Evangelists would be responsible for spreading the meme. In particular I would really like the evangelists to come up with ways how to intuitively communicate the Level 3 issues to Level 1 and Level 2 people. Things like this marvelous picture here. [image of a polar bear on a block of ice amid clear water] It illustrates an abstract issue of global warming on a level where instincts can really engage with it.

Of course, the movement should not be limited to the roles and ideas presented here. For example, here is an idea that came up in a discussion with Nick Bostrom. What if we set up and promoted a fund that would pay out in the future to the people who in retrospect have made significant contributions from the perspective of future society? We might as well call this fund the Petrov fund, because we desperately need more people like him.

So, as Max Tegmark said yesterday, in conclusion, the future of the world and indeed the universe no longer depends on societies. It depends on people with red buttons, and to navigate this new world, we need people who distrust their instincts, and rely on reason instead. We need the CL3 generation.

Finally, since I know that our instincts don’t care about saving the universe, let me just point out the universe contains our children. So what this talk was really about was saving the lives of the children. Your children. Thank you. [applause]

[Q&A begins]

Man 1: You mentioned the social status reinforcement cycle, and that plays on the idea that culture acts as a feedback mechanism into the actual outcomes that you get on addressing these challenges. There was a quote I heard recently “You get what you celebrate,” so what we recognize in the social status feedback mechanism sort of determines the outcome. How much of a role do you see that the overall culture and values of society have on the outcomes that we get on addressing these challenges?

Jaan Tallinn: It depends on what you mean by “us.” Us as a civilization, they really influence us, they really bog us down. That’s what I mean. I see two sides to the coin. One is that, like I said, the future is increasingly determined by individuals, but that actually can also be a good thing, because in the context that the vast majority of people do not really understand that something has changed this century, we have a chance to promote those people who do understand into the position where they are they key to the future.

[next question]

Man 2: You mentioned that the evolution of the trichotomy has come about because of technology basically producing a qualitative difference between society and future society, and that therefore there’s a discontinuity because of the lack of cultural memory that produces the linkages between those two. And then you fast-forwarded to the end where you said for example that you turned down a museum appointment because that was a Level 2 consideration as opposed to a Level 3. But earlier on, you also talked about your own personal reflections which talks about your own progression from not really feeling that you’re going to have an impact to obviously then having an impact. So you obviously yourself have a personal cultural awareness. Is the fact that the people who are working on Level 3 challenges without actually having an understanding or an appreciation of the Level 2, exactly what then results in this discontinuity of the trichotomy? And people who have an organic understanding of Level 2, and informing that in Level 3 is probably more hopeful than people divorcing themselves from Level 2 and introducing themselves straight into Level 3.

Jaan Tallinn: Yes, I think I made this point three times in the presentation. I don’t discount people who really work on Level 1 or Level 2, and I acknowledge the problem that you point out. Especially because the generation is really young, they don’t really have much accomplishments to show for, so if you are a Level 3 person that doesn’t have much accomplishments, you find it really really hard to be taken seriously. Therefore I find myself in a really interesting unique position where I can actually... Yes, exactly, thanks to my past, I have enough Level 2 currency to apply towards making people aware of Level 3 issues.

[next question]

Man 3: As a Level 1 person, I would like to know what software did you use to make your presentation? [laughter]

The MC: It’s called Prezi.

Man 3: Thank you very much.

Jaan Tallinn: It’s my first presentation with that. It has its quirks, but it’s good.

[next question]

Man 4: Related to what you said a couple question ago, your own progression from Level 1 to Level 3, you know some people do that because they see their children. Erik Erikson would say that you’re at the life stage where you need to reach out. You also have the money and the wherewithal now to do that. What do you think accounted for that epiphany for you to go from Level 1 to Level 3?

Jaan Tallinn: It was the writings of Eliezer Yudkowsky that opened my eyes. That’s the simple way of putting it. I always cite Moshe Looks, when he went to an intelligence conference in Oxford said that “You know, you really should listen to Eliezer Yudkowsky, even if it’s annoying to admit that he’s right.” [laughter]  [applause]