AI

"A great design appears at first insane;
But chance will soon seem quaint and blind,
And such an exemplary thinking brain
Will soon by thinkers be designed"
- Goethe, ‘Faust’

Start here

• Read: The Artificial Intelligence Revolution -  Wait But Why. Introductory post summarizing the potential dangers of Artificial General Intelligence  
• Listen: Sam Harris & Eliezer Yudkowsky - Waking Up. Good intro to AI safety
• Do: AIsafety.com - virtual reading list on AI safety, meets every Wednesday.
• New: Malicious Use of AI Report Miles Brundage et al.

Summary

Intro

• 71 Definitions of Intelligence - Shane Legg. Also: What is intelligence? Luke Muehlhauser on intelligence as optimization power divided by resources used.
• The Artificial Intelligence Revolution -  Wait But Why. Introductory post summarizing the potential dangers of Artificial General Intelligence  
• Superintelligence - e-book by Nick Bostrom. Classic on Superintelligence, that explores different take-off scenarios of AI, why they’re dangerous and urges for preparatory action now.
• That Alien Message - Eliezer Yudkowsky. Challenges mainstream human intuitions on speed of AI.
• Sam Harris & Eliezer Yudkowsky - good intro on common problems in AI safety

Progress & Speed

• Slow Takeoff vs Fast Takeoff - Paul Christiano
• The AI Foom Debate between Yudkowsky & Hanson. Hard or slow take off? Also as video here: Yudkowsky vs Hanson: Singularity Debate. As background: Intelligence Explosion Microeconomics - Eliezer Yudkowsky. Suggests a methods to test whether a sufficiently advanced machine intelligence could build a smarter version of itself,
• which could in turn build an even smarter version, and that this process could continue to the point of vastly surpassing human capability.

Risks

• Malicious Use of AI Report Miles Brundage and many other authors
• AI as positive and negative factor in Global Risk - Eliezer Yudkowsky
• There is no fire alarm for AI safety - Eliezer Yudkowsky
• Many chapters in Rationality: From AI to Zombies discuss AI Risk. Especially good are My Naturalist Awakening, That Tiny Note of Discord, Sympathetic Minds, Sorting Pebbles into Heaps, No Universally Compelling Argument, The Design of Minds in General, Detached Lever Fallacy, Ethical Injunctions, Magical Theories, Fake Utility Functions.
• Orthogonality Thesis - Bostrom, Sandberg, Douglas. Why Intelligence and goals are orthogonal. Also: Basic AI drives - Steve Omohundro. sufficiently advanced AI systems of any design would, by default, have incentives to pursue a number of instrumentally useful subgoals, such as acquiring more computing power and amassing many resources, which are harmful even if the overarching goals are not.

Safety

• AI Safety: Why It’s Hard & State of the Art. I give an overview of AI safety and why the field can be broken down into the four focus areas below (Ethics, Technical Alignment, Cybersecurity, Social Coordination), including a brief overview of the readings in those areas
• The Landscape of AI Safety and Beneficence Research: Input for Brainstorming at Beneficial AI 2017, Richard Mallah.
• Long-term strategies to end existential risk from hard take-off - Daniel Dewey. Covers 4 scenarios: International coordination, sovereign AI, AI-empowered project, Other decisive technological advantage
• Beneficial AI Conference -good Youtube channel by FLI
• 3 Principles for Safe AGI - TED talk by Stuart Russell
• Decentralized Approaches to Reducing Existential Risks - Mark Miller, Christine Peterson, Allison Duettmann on intelligence as problem-solving ability of ecosystems

Ethics

• Coherent Extrapolated Volition - Eliezer Yudkowsky. Good paper to understand why AI safety is hard. Objections to Coherent Extrapolated Volition -Lesswrong.
• Value Alignment Landscape - Lucas Perry. Interactive, comprehensive overview of the field of value alignment.

Technical Alignment

• AI Safety Research Landscape - FLI project on mapping AI safety research.
• AI Alignment: Why it’s hard and where to start - Eliezer Yudkowsky. Includes a nice overview of AI alignment
• AI Alignment -Paul Christiano’s website on AI alignment. Especially Directions and desiderata for AI alignment - Paul Christiano
• Ensuring Smarter-Than-Human Intelligence Has a Positive Outcome talk by Nate Soares
• Research Priorities for Robust AI - Stuart Russell, Max Tegmark, Daniel Dewey
• Intelligence Distillation  and Reframing Superintelligence - Eric Drexler

Social Coordination

• Reading Guide for the Global Politics of Artificial Intelligence - comprehensive reading list by Allan Dafoe.

• Smart Policies for Artificial Intelligence - Miles Brundage, Joanna Bryson
• A Survey of Artificial General Intelligence Projects for Ethics, Risk, and Policy, Seth Baum
• Policy Desiderata in the Development of Machine Superintelligence, Future of Humanity Institute, 2017.        - Bostrom, Nick, Dafoe, Allan, Flynn, Carrick.

• AGI Timelines & Policy White paper Allison Duettmann.
• Deciphering China’s AI Dream, Jeffrey Ding
• Avoiding the Precipice: Race Avoidance in the Development of AI, Olga Afanasjeva, Jan Feyereisl, Marek Havdra
• Racing to the precipice: A model of artificial intelligence development, Stuart Armstrong, Nick Bostrom, Carl Schulman        
• Beyond MAD?: The Race for Artificial General Intelligence - Roman Yampolskiy        
• AI And The Future of Defense - Stephan De Spiegeleire, Matthijs Maas, Tim Sweijs
• Governing Boring Apocalypses: A New Typology of Existential Vulnerabilities and Exposures for Existential Risk Research, Futures, 2018 - Liu, Hin-Yan, Cedervall Lauta, Kristian, Maas, Matth
• Unilateralist’s Curse: The Case for a Principle of Conformity by Anders Sandberg, Nick Bostrom, Tom Douglas
• Strategic Implications of Openness in AI Development, Nick Bostrom        

• Livestream of seminar on Artificial General Intelligences & Corporations @Internet Archive  and Transcript of seminar on AGI & Corporations @ Internet Archive

Hope

• Raikoth - Scott Alexander
• 12 AI Aftermath Scenarios - Max Tegmark

Action

News

• AIsafety.com - virtual reading list on AI safety, meets every Wednesday
• EFF AI Metrics - tracking AI progress and development, updated regularly
• Machine Ethics index - Nell Watson. Literature index on machine ethics
• Guide to working in AI policy - Miles Brundage.

AI safety orgs

• pretty comprehensive list by Future of Life Institute on AI safety orgs
• Partnership on AI - Established to study and formulate best practices on AI technologies, to advance the public’s understanding of AI, and to serve as an open platform for discussion and engagement about AI and its influences on people and society.Established to study and formulate best practices on AI technologies, to advance the public’s understanding of AI, and to serve as an open platform for discussion and engagement
• Machine Intelligence Research Institute - MIRI's artificial intelligence research is focused on developing the mathematical theory of trustworthy reasoning for advanced autonomous AI systems.
• OpenAI - OpenAI is a non-profit artificial intelligence research company that aims to promote and develop friendly AI in such a way as to benefit humanity as a whole
• DeepMind - AI company, bought by Google. Has strong safety focus. Now established Deepmind Ethics & Society Unit
• Center for Human-Compatible AI - CHAI's goal is to develop the conceptual and technical wherewithal to reorient the general thrust of AI research towards provably beneficial systems.
• Leverhulme Centre for the Future of Artificial Intelligence - A global community to ensure that AI benefits all of humanity
• Future of Humanity Institute Future of Humanity Institute (FHI) is multidisciplinary research institute working on Existential Risk at the University of Oxford.
• Future of Life Institute - a volunteer-run research and outreach organization in the Boston area that works to mitigate existential risks facing humanity, particularly existential risk from advanced artificial intelligence
• Foresight Institute - Foresight Institute is a leading non-profit research organization focused on technologies of fundamental importance for the human future, focusing on molecular machine nanotechnology, cybersecurity, and artificial intelligence.

Cyberspace

Governments of the Industrial World, you weary giants of flesh and steel, I come from Cyberspace, the new home of Mind. On behalf of the future, I ask you of the past to leave us alone. You are not welcome among us. You have no sovereignty where we gather.

• JPB, Declaration of Independence of Cyberspace

Intro

• Declaration of Independence of Cyberspace - John Perry Barlow. Speech given by JPB at the World Economic Forum on the properties of cyberspace that make it unlike any other invention.
• Transparent Society - David Brin
• Computer Lib - Ted Nelson.
• Society of Mind - Marvin Minsky..
• A Hacker Manifesto - McKenzie Wark. criticizes the commodification of information in the age of digital culture and globalization
• Agoric Papers - Eric Drexler, Mark Miller. Three papers that became classics for their treatment of computation as an economic system.
• Opensource Hardware Design Principles - guarding privacy, avoiding trap doors. Purism - open-source hardware design company
• Cyberinsecurity As Catastrophic Risk - notes on strategy session at 2017 Vision Weekend, held by Mark S. Miller
• Cyber War - Richard Clarke.
• The Stack - Benjamin Bratton. He proposes that different genres of planetary scale computation -smart grids, cloud platforms, mobile apps, smart cities, the Internet of Things, automation- can be seen as forming a coherent whole: an accidental megastructure that is both a computational infrastructure and a new governing architecture.
• Symbiot Papers - Three papers on cybersecurity offense and evolutionary inspired risk models and architectures

Cybersecurity & AI

• AGI Timelines & Policy - go to section on cybersecurity - Allison Duettmann. Whitepaper based on private meeting on AGI safety in 2017
• Cyber, Nano, and AGI Risk - and Decentralized Approaches to Reduce Risk - Christine Peterson, Mark Miller, Allison Duettmann. Cybersecurity as near-term, undervalued catastrophic risk that is more imminent than risk from AGI and potential solutions.
• AI and Cyber security - Roman Yampolskiy. Cybersecurity approaches for AI safety
• AI Alignment & Security - Paul Christiano
• The Malicious Use Of Artificial Intelligence: Forecasting, Prevention, and Mitigation -  go to section on cybersecurity. Strongly recommend this paper, published by FHI, written by Miles Brundage Shahar Avin Jack Clark Helen Toner Peter Eckersley Ben Garfinkel Allan Dafoe Paul Scharre Thomas Zeitzoff Bobby Filar Hyrum Anderson Heather Roff Gregory C. Allen Jacob Steinhardt Carrick Flynn Seán Ó hÉigeartaigh Simon Beard Haydn Belfield Sebastian Farquhar Clare Lyle Rebecca Crootof Owain Evans Michael Page Joanna Bryson Roman Yampolskiy Dario Amodei

Computation

Quantum Computing

• The Arrival of Quantum Computing - Will Zeng
• Steve Jurvetson on Quantum Computing - After-On interview with Steve Jurvetson on Quantum Computing

Action

• Electronic Frontier Foundation - The Electronic Frontier Foundation (EFF) is an international non-profit digital rights group based in San Francisco, California.
• Data & Society - Data & Society is a research institute focused on the social and cultural issues arising from data-centric technological development.
• Internet Archive - Internet Archive is a non-profit library of millions of free books, movies, software, music, websites, and more
• Center for Long-term Cybersecurity - is developing and shaping cybersecurity research and practice based on a long-term vision of the internet and its future

What next?

• for a reminder of why to care, see Existential Angst & Existential Hope
• for getting to work, see Effective Altruism
• for neurotechnology & AI safety, see Living Longer, Better, Smarter
• go back to the overview at Existentialhope.com