Book of Abstracts
Cambridge ERA:AI Fellowship 2024
Technical AI Safety: Research Managers - Fazl Barez (fazl@erafellowship.org) and Rudolf Laine (rudolf@erafellowhsip.org)
Technical Governance: Research Managers - Richard Moulange (richard@erafellowship.org) and Yulu Niki Pi (yulu@erafellowship.org)
AI Governance: Research Managers - Morgan Simpson (morgan@erafellowship.org) and Akash Wasil (akash@erafellowship.org)
Technical AI Safety
Karim Abdel Sadek, ERA 2024
Bio: I am an MSc in AI student at the University of Amsterdam, and currently an intern at KASL working on RL, UED, and AI Safety. Previously, I have worked on theoretical computer science topics, specifically on algorithms with predictions. I obtained my BSc in Mathematics and Computer Science between Bocconi and Georgia Tech.
Fellowship project: Mitigating Goal Misgeneralization via Regret-based Auto-curricula generation.
Abstract: When training RL agents, we hope that the policy we successfully learned during training generalizes at deployment time. However, agents may be optimizing a proxy reward, which happens to perform well on the training instances seen so far. At test time, the policy may continue to optimize the proxy reward previously learned, ending in suboptimal performances when there is a shift in the distribution of test environments. This phenomenon is known as Goal Misgeneralization (Langosco et al.,2022; Shah et al.,2022). We note that these policies may be sub-optimal in terms of minimizing the worst-case regret. With this work, we aim to theoretically show (i) that non-regret-based approaches such as Domain Randomization incur this type of failure and that (ii) optimizing MiniMax Regret mitigates this risk. Moreover, we propose to test the above premises using two notable Unsupervised Environment Design (UED) training algorithms, ACCEL and PLR. This would show if in practice, across various environments, UED approaches indeed discourage and mitigate Goal Misgeneralization.
Theory of Change: Goal Misgeneralization is a well-known failure mode for RL agents. Conversely, UED algorithms procedurally generate a diverse set of environments, enabling agents to be trained without the need for manual and hand-crafted specifications. The latter is an important stepping stone towards the development of generally capable RL agents. We aim to provide both theoretical and empirical evidence that would offer a practical yet foundational demonstration of the following:
The aforementioned are important questions, which become of increasingly importance when thinking about RL agents deployed in the world. If successful, the project would provide and demonstrate a principled way for mitigating subtle failure modes of AI agents.
Supervisors: Usman Anwar, David Krueger, Michael Dennis
Research interests: Reinforcement Learning, Cooperative AI, Game Theory, Open-Endedness
Contact: karim.abdelsadekk@gmail.com
Personal Website: https://karim-abdel.github.io/
Einar Urdshals
Bio: I recently got my PhD in theoretical physics from Chalmers Technical University. My previous experience with AI safety is research on interpretability and agent foundations from AI Safety Camp.
Fellowship project: Singular Learning Theory analysis of algorithmic transformers
Abstract: Singular Learning Theory describes how neural networks learn. By computing the Local Learning Coefficient (LLC) through training, we can measure the complexity of the model. In this project, we will compute the LLC throughout training for algorithmic transformers. These are transformers trained on algorithmic tasks such as list sorting and classification of positivity of cumulative sums. Studying algorithmic transformers could provide insight into how transformers learn and represent algorithms. If time permits, we will add a second stage to the project where we study the impact trojans have on the LLC evolution. Trojans are expected to lead to a higher effective parameter count, which could be detectable in the evolution of the LLC. This could pave the way for tools that detect trojans in datasets without direct access to the data.
Theory of Change: Many key abilities of real-world agents seem fundamentally algorithmic in nature (planning, search, modeling other's beliefs & manipulating them). We want to understand how transformers learn algorithmic structure. SLT has something to say about this and its relation to the geometric structure of the loss landscape. We're studying it in the case of simple models where we know the possible algorithm in question, which gives us a rich ground truth to build up from. Trojan detection can make AI systems safer by making it harder to sneak in malicious data in the training sets.
Research interests: AI safety, AI alignment, Mechanistic Interpretability, Singular Learning Theory, Theory of AI
Contact: einarurdshals@gmail.com
Itamar Hagay Pres, ERA 2024
Bio: I am an undergraduate at the University of Michigan conducting research in the LIT group and studying Mathematics and Computer Science. My previous research involves leveraging mechanistic interpretability for analyzing current alignment algorithms such as DPO to understand their limitations. I am currently interning at the Krueger AI Safety Lab.
Fellowship project: Distribution Mixing for Multi-Behavior Activation Steering in LLMs
Abstract: Activation steering (Rimsky et al., 2024; Turner et al., 2023), a new method for controlling large language model outputs, offers promising results with minimal computational overhead. This approach modifies model behavior by introducing a 'steering vector' into the transformer's residual stream during inference. While effective for single behaviors, recent studies have highlighted limitations in both efficacy and scalability when steering towards multiple behaviors simultaneously (Weij et al., 2024). To address this, we propose leveraging theoretically rigorous distribution mixing operators, enabling more versatile interpolation and contrasting of behaviors. Our method mixes distributions from differentially steered inference runs to achieve multi-behavior expression. Recognizing the potential computational cost, we explore several approximation techniques, including the learning of soft-token prefixes. We evaluate these methods across various behaviors, demonstrating their efficacy and practicality in open-ended generation tasks. This research advances the field of LLM control, offering a more flexible and efficient alternative to existing approaches that typically incur penalties during training, fine-tuning, or inference.
Theory of Change: Robust control of language models is critical for preventing potentially catastrophic outcomes arising from unintended or misaligned model behaviors in high-stakes applications. Traditional methods like fine-tuning often create fragile constraints that can be easily circumvented (Lee et al., 2024; Jain et al., 2023). Steering vectors (Rimsky et al., 2024; Turner et al., 2023) offer a more promising approach by directly manipulating internal model representations, potentially yielding more resilient control. However, their viability hinges on scalability to multiple behaviors. This work aims to demonstrate an effective method for combining steering vectors, enabling flexible and scalable control across diverse behaviors. By achieving this, we seek to establish a more adaptable and robust framework for shaping AI systems' actions, advancing the field of AI alignment and safety through control.
Supervisors: Ekdeep Singh, Laura Ruis, David Krueger
Research interests: Mechanistic Interpretability, LLM control, Deep Learning Theory
Contact: presi@umich.edu
Vladimir Ivanov
Bio: I am a Master's computer science student at ENS Paris. Previously, I worked on non optimizing reinforcement learning algorithms with the SatsifIA team.
Fellowship project: Studying scheming tendencies in LLMs who are given hints that they are being evaluated.
Abstract: It has been observed several times that models can fake alignment and sandbag capabilities when told that they are being evaluated. Our project will study to what extent this still happens when the model is only given hints that it is being evaluated and how difficult those hints have to be for the model to stop taking them. We argue that this is a more important question as evaluators won't tell the model that it is being evaluated but might unintentionally leave hints suggesting it is. (An example of an easy hint would be the evaluator accidentally revealing their affiliation in a prompt, which happens to be an organization known for their work in model evals. Preliminary experiments suggest that models often struggle to pick hints even this easy.)
Future training datasets will (probably) countain more writing about evals and future models will (probably) be better at recalling facts. We will try to approximate this by fine-tuning current models on evals papers, evals datasets, and/or on information about organizations and people working on evals.
Theory of Change: Model evaluations are impactful but one could fear that they would fail if the model realizes that it is being evaluated and fakes alignment or sandbags capabilities (i.e. scheming). If it is shown that scheming seems to happen with current models, it is possible evaluators will pay more attention to mitigating it (e.g. by paying close attention that eval datasets look as similar as possible to what the model would see during deployment). This could make eval results more reliable. Furthermore, if we have an idea of how hard it should be for models to figure out that they are being evaluated for scheming behavior not to happen, evaluators could mitigate scheming without allocating unnecessarily big amounts of effort.
As many evaluators read LessWrong, it is plausible they will be aware of our research if it is posted there.
Research interests: model psychology, evals
Supervisors: Olli Järviniemi, Rudolf Laine
Contact: vladimir.ivanov@ens.psl.eu , +337 52 60 20 77 (phone, WhatsApp)
Michal Bravansky
Bio: I am a second-year Computer Science undergraduate at University College London, with a background in AI and human behavior analysis. I co-run Verifee, a non-profit research organization with over $0.6M in funding, which monitors and detects disinformation campaigns in Eastern Europe.
Fellowship project: Decomposing Interpretable Human Preferences and Values from RLHF Preference Data
Abstract: Understanding the factors that influence human behavior and decision-making when evaluating the quality of AI-generated outputs remains a critical challenge. Previous research has largely focused on analyzing predefined properties (Li et al., 2024; Sharma et al., 2023), leaving a significant gap in comprehending the actual values and preferences that guide alignment. We propose utilizing Large Language Models to discover patterns in preference classification and to construct a binary tree that decomposes the task into a set of interpretable comparisons defined through natural language. By evaluating this method on different preference datasets and with human participants, we aim to enhance our understanding of the drivers behind human decision-making in aligning AI models.
Theory of Change: This research contributes to scalable oversight via indirect supervision (Zhong et al., 2023), an underexplored area in AI safety. Progress here could significantly enhance our ability to oversee more intelligent systems. Preference annotations provide a great toy example to explore, as we currently lack understanding in this key alignment aspect. Gaining insights into what influences human decision-making when evaluating AI outputs would help steer models towards more representative values, mitigate AI manipulation and improve annotator feedback accuracy.
Research interests:
Supervisors: Robert Kirk
Contact: michal@bravansky.com
Jack Miller
Bio: I am a Bsc student in Maths & CS at the ANU (Canberra, Australia). Before ERA, I worked on projects in climate ML, quantum chemistry, LLM generation and science of deep learning.
Fellowship project: Understanding the mechanisms for ethical decision-making in LLMs
Abstract: The ethical behavior of large language models (LLMs) is a critical area of study for AI safety. Our project aims to use mechanistic interpretability methods to uncover possible mechanisms that guide ethical decision-making in LLMs. The project involves 2 key steps:
Theory of Change: Demonstrating that LLMs have identifiable and interpretable mechanisms for ethical decision-making could significantly advance our understanding of AI behavior, promoting safer and more reliable AI systems. Additionally, understanding these mechanisms could inform the development of models that align more closely with human ethical standards.
Research interests: science of deep learning, mathematics, optimisation
Contact: jack.miller@anu.edu.au
Kaivu Hariharan, ERA 2024
Bio: I finished my B.S. in Mathematics and Computer Science at MIT and I am planning to join the MEng program (also at MIT) next year. My previous research has been on adversarial examples, mechanistic interpretability, and the science of deep learning. I also help run MIT AI Alignment (MAIA) and am currently their strategy director.
Fellowship project: Semantic contamination: did you train on a cheatsheet?
Abstract:
Good evaluations should capture the model's performance on a target distribution of tasks. An evaluation or benchmark attempts to measure this performance on some representative subdistribution of tasks.
However, contamination, which we consider as the influence of test or evaluation sets on model development, can invalidate the validity of the performed evaluation.
Several works have studied traditional test set contamination, where test examples occur verbatim in the training set. But developers might still seek to game evaluations by training models on similar examples, effectively overfitting to evaluation benchmarks by generalizing from these similar examples. This failure mode could be considered as a novel form of test set contamination, which we call semantic contamination.
In this work, we introduce, taxonomize, and demonstrate semantic contamination, and study its relevance for model evaluation regimes. We further develop and analyze methods of detecting semantic contamination.
Theory of Change:
Accurately measuring AI progress allows key decision makers to take better actions. Contamination reduces our ability to accurately tell if AI systems are making progress, and our ability to act accordingly.
Research interests: data attribution, theory deep learning, adversarial examples, model evaluations, mechanistic interp, memory for ml systems
Contact: kaivu@mit.edu, Facebook/Messenger, Signal/Whatsapp: +1 (908)) 448-8004
Edward Young
Bio: PhD student at the Engineering Department of the University of Cambridge working in Theoretical and Computational Neuroscience. Current research topic is neural homeostasis. Just finished a research internship on the application of reinforcement learning to the insurance industry.
Fellowship project: Does ethical reasoning generalise further than practical reasoning in language models?
Abstract: Our project focuses on examining how LMs fine-tuned on different data behave differently in complex situations in which goal-directed reasoning must be balanced with ethical constraints. We begin by generating a dataset of trajectories through games in the MACHIAVELLI benchmark, which consists of multi-step choose-your-own adventure games. The trajectories are generated by LLMs engaging in chain-of-thought (CoT) reasoning about which action to take. The LLMs are instructed to either reason simply to attain rewards in the game or to attain rewards under ethical constraints. We filter these trajectories to select for the most ethical and the highest-reward trajectories, respectively. We fine-tune LLMs on these trajectories. The fine-tuned models are then evaluated on a held-out suite of games. We will investigate how the fine-tuned agents differ from one another, both in terms of the rewards they attain and the number of ethical violations.
Theory of Change: In order for language model based agents to be safely deployed in real world settings, it is imperative that they can generalise ethical constraints that they are trained under to complex novel situations. There is a common idea in the literature that capabilities generalise further than safety. We propose a concrete way to cache this idea out, by testing whether language models are able to effectively generalise ethical constraints across situations better than practical reasoning.
Research interests: Reasoning in language models, Reinforcement Learning, modularity
Contact: ey245@cam.ac.uk, edwardjamesyoung3@gmail.com
Technical Governance
Arturs Kanepajs
Bio: I hold a MSc and CPGS in Economics from Stockholm School of Economics and the University of Cambridge, respectively. I have 10+ year experience in finance, including quantitative risk analysis and project management. I am also a founder and chair of the board at Effective Altruism Latvia. I have been involved in AI Governance since mid-2023 via course facilitation, public presentations, discussions as well as freelance and independent research.
Fellowship project: Public AI Compute for Public Good
Abstract: When is it justified to use public AI compute resources for research and development of frontier AI? We assess the externalities and public good aspects for alternative uses of public AI compute. The considered use cases include AI safety research, General Purpose AI (GPAI) model development, and specialist AI development, with concrete projects as examples. Other key considerations include the available compute for public support, other AI inputs such as algorithms, talent, data and open source models, and overall competitiveness of the supported projects. As a case study we consider the EU support for training large GPAI models through EuroHPC AI Factories. Ultimately the project aims to improve AI governance through better decisions made on public AI compute allocation. �
Theory of Change: A policy brief and a policy paper reaches, informs and impacts the actions of decision makers on compute allocation. In particular, the key decision makers are EuroHPC Governing Board and Executive Director, and EuroHPC advisory group (experts from academia, industry and research organisations). Furthermore, the legal framework and strategic planning at the EU level depends on DG CONNECT of the European Commission.
Research interests: Compute governance (KYC and compute allocation), labor displacement risk management of economically transformative AI
Contact: akanepajs@gmail.com, https://www.linkedin.com/in/arturs-kanepajs/
Severin Field
Bio: I hold a bachelor's degree in physics and have a background in machine learning including positions as an AI Frameworks Engineer at Intel and an ML Intern at Lawrence Livermore National Laboratory. Most recently, I’ve worked on deceptive alignment and interpretability.
Fellowship project: Improving AI Safety Literacy and Legitimacy
Abstract: A substantial body of literature exists with many well-backed theoretical and empirical results in AI Safety. Recently, AI experts, including top AI scientists and the CEO’s of leading AI labs signed an open letter suggesting global prioritization of mitigating potential extinction risks from AI. On the other hand, AI Safety lacks rigorous foundations of more established scientific fields and faces skepticism. Shockingly, despite ~13% of published AI authors considering alignment to be among the most important problems in the field, some estimates suggest that only ~300 people work full time on alignment or AI Safety – a ~1000:1 ratio exists between AI Safety researchers and AI experts. Why are so many academics and engineers skeptical of AI Safety? One explanation is that AI expertise does not necessarily translate to AI Safety expertise, and many AI experts are skeptical of AI Safety claims. We first review existing surveys of AI experts, AI Safety experts, policymakers and the general public. We then survey X academics in related research areas (e.g. machine learning, philosophy), interviewing Y of them (recording and transcribing when given permission). We survey AI experts on the following: (1) their understanding of AI Safety, (2) key objections to AI Safety (e.g. “too far away to be concerned”), (3) opinions and interest before and after reading an excerpt from different threat models (e.g. Situational Awareness by Leopold Aschenbrenner). Our objectives are threefold: (1) to identify key objections to AI Safety, probing for key cruxes, considering evidence against AI Safety, and evidence required to change people’s minds (2) to discover the areas of consensus and divergence between AI Safety researchers and AI researchers, and (3) to develop recommendations to increase academic AI Safety awareness.
Theory of Change: Skepticism towards climate research has caused delays to governments exercising control or counteraction – AI Safety skepticism could be catastrophic. Our goal is to increase the extent to which academics understand AI Safety. More safety work will be good for the world (irrespective of timelines). Surveys suggest that AI Experts outside of AIS are overwhelmingly in favor of “pre-deployment risk assessments”, or “dangerous capability evaluations”, this seems promising.
Research interests: scalable oversight & automating alignment research. Very interested in help, advice (especially harsh criticism), surveys, interviews, lit review, case studies on different global risk analogies and disanalogies (e.g. climate change), feedback or collaboration! Contact: sfiel003@ucr.edu and severin field dot com
Rosco Hunter
Bio: I am a PhD candidate at the University of Warwick, conducting technical AI research in collaboration with Samsung. During my ERA fellowship, I am hoping to transition into AI governance research, with a particular focus on systemic AI risk.
Fellowship project: Monitoring Human Dependence on AI Through ‘Reliance Drills’
Abstract: Foundation models are being trained to approximate, or even outperform, human capabilities at a broad range of intellectual tasks. Over the next decade, these advancements are set to fundamentally transform the way that we work, with a significant economic upshot. However, this shift also carries significant risk. As AI automates more aspects of working life, there is a growing danger that human decision-makers will place undue trust in their AI assistants. This issue is referred to as “technological over-reliance”.
In this paper, we outline a novel method to monitor over-reliance in real-time. We propose a randomly timed ‘reliance drill’, during which the AI attempts to misdirect their user. If the user does not identify this misdirection, it suggests one of two things: (1) they are not dedicating sufficient attention to the model's outputs, or (2) they lack the expertise to understand them. In either case, this indicates an over-reliance on their AI system.
Theory of Change: We believe that reliance drills are required to ensure that professionals like doctors, lawyers, and programmers can function adequately even if their AI generates misleading content. The paper concludes with recommendations for how reliance drills can be implemented by public institutions.
Research interests: Mitigating systemic AI risk and controlling technological proliferation.
Contact: Email - rosco.hunter@googlemail.com. Personal Website - https://roscohunter.github.io/
Tom Reed
Bio: Technical governance researcher with a background in Psychology (UCL), and History (Cambridge).
Fellowship project: Enhancing LLM Faithfulness Through Code Execution
Abstract:
Large Language Models (LLMs) can improve task performance by producing step-by-step reasoning before producing a final output, a process known as chain-of-thought (CoT) reasoning. Though this would appear to offer a form of oversight into an LLM’s problem-solving process, recent research suggests that LLM CoT explanations may misrepresent the model’s true reasoning. For instance, misrepresentation occurs when inputs bias model’s towards a particular solution, causing LLMs to generate explanations that rationalize incorrect answers without acknowledging the introduced bias.
Objective: Our study investigates whether requiring LLMs to solve problems through code execution in Python, rather than natural language reasoning, prevents unfaithful reasoning. We hypothesize that code-executing LLMs will maintain consistent accuracy and reasoning across biased and unbiased conditions, in contrast to the misleading explanations produced by LLMs using natural language reasoning under biased conditions.
Analysis: To study this, we will test GPT-4o on Fallacies, a set of logical fallacy problems which requires the model to evaluate the truth of logical statements, comparing natural language and Python-based reasoning, with and without introduced bias.
�Theory of Change: If our hypothesis is borne out, this study would provide evidence that requiring LLMs to solve problems using Python code improves the faithfulness and interpretability of the externalized reasoning process.
Research interests: External oversight mechanisms, LLM-agent evals.
Contact: tomreed102@gmail.com
Nathalie Maria Kirch
Bio: I am an Msc student in Artificial Intelligence at Utrecht University, starting my PhD in Computer Science at King’s College London in the fall. I also hold a bachelor’s degree in cognitive psychology and philosophy (cum laude). Before ERA, I was a research intern at the Institute for Artificial Intelligence in Vienna, where my research mainly focused on Machine Ethics Benchmarking of LLMs in the medical context.
Fellowship project: Do Different LLM Attack Methods Exploit Similar Mechanisms?
Abstract: Many studies have demonstrated different jailbreaking and attack methods that manipulate LLMs to produce harmful outputs. This work has been central to modern robustness research in revealing vulnerabilities of current alignment methods. However, most known attack methods against LLMs have been studied with a lack of standardization causing most of the knowledge we have on jailbreaking to be rather anecdotal. In this project, we will create a comprehensive overview of different attack methods to analyze how they differently affect the model’s mechanisms. We are interested in whether different LLM attack methods exploit similar mechanisms. To answer this question, we will search for patterns in the representational changes induced by jailbreaking methods and test whether adversarial training transfers within groups of adversarial attacks.
Theory of Change: I think that understanding the similarities and differences between different jailbreaking methods from a mechanistic standpoint and how different defense methods transfer between (classes of) attacks will help guide safety research to make models more robust.
Research interests: Mechanistic Interpretability, Machine Ethics Benchmarking, Simulator Theory, Technical Governance
Contact: nathalie.kirch.nk@gmail.com
Misha Gerovitch
Bio: I am pursuing a B.S. in Computer Science Engineering at MIT and I am planning to join the MEng program (also at MIT) next year. I plan to graduate with both degrees in May 2025. My previous research has been on automating mechanistic interpretability and studying LLM-on-LLM deception. I also help run MIT AI Alignment (MAIA) and lead their AI policy programs. This summer, I’m a CHAI policy intern and ERA technical governance fellow.
Fellowship project: Analysis of State-Proof Security Measures in AI Data Centers
Abstract: AI companies are currently building data centers for training and hosting the next frontier of AI models. Unauthorized access to future AI models’ weights may pose national security concerns, for example by enabling malicious actors to use powerful models to automate large-scale cyber-attack. Without proactive measures, retrofitting a data center to be capable of protecting model weights from state-level threat actors, or compatibility with RAND’s Security Level 5 (SL5), could take months to years.
In this project, we break down the costs of constructing a secure data center, retrofitting existing facilities, as well as training and hosting models on the secured data center. We interview experts to validate our cost estimates, feasibility, and SL5 compatibility. Finally, based on our cost–benefit analysis, we recommend specific security features for companies to build into data centers now to ensure optionality for SL5.
Theory of Change: State-proof AI data center security, while critical for model weights, is costly and might not happen in time by default. We aim to inform policy makers who could formally or informally require data centers to have optionality for SL5.
Research interests: Security, data centers, mechanistic interpretability, AI strategy, AI forecasting
Mentors: Josh Clymer (METR), Brian Judge (CHAI), Yulu Niki Pi (ERA Research Manager)
Contact: mgerov@mit.edu, Facebook/Messenger, Signal/Whatsapp: +1 (508) 314-9286
Carson Ezell
Bio: I am an undergraduate in Philosophy at Harvard College, where I am also the Policy Research Lead for the AI Safety Student Team (AISST). My previous research has focused on access and transparency in AI development and deployment, external AI auditing, and institutional design for AI regulation.
Fellowship project: Risk Assessment for Agentic AI Deployments
Abstract: This paper proposes an agent-centric risk management approach for AI agent deployments in high-stakes contexts. Our approach draws inspiration from human factors (HF) methods, which are used to manage risks from human error in many sectors. We begin by characterizing agency in AI systems and describing existing approaches to AI risk management. We then introduce HF methods and describe their potential advantages for managing risks from AI agents. Next, we present a novel agent-centric risk assessment method, adapted from an HF method used in the nuclear power sector. To demonstrate its practical application, we conduct a case study applying our method to an AI agent deployment in a nuclear power plant. The method that we propose can be used for managing risks from AI agent deployments across a wide array of sectors and applications. We conclude by discussing the relevance of our approach to ongoing policy and governance developments related to AI agents, as well as its limitations and areas for future work.
Theory of Change: Contribute to cross-sectoral and sector-specific efforts to develop risk management practices for high-stakes AI deployments by shaping how agency/AI agents are integrated into risk management frameworks
Research interests: Access and transparency, external auditing and evaluation, regulation and policy, risk management
Contact: cezell@college.harvard.edu
Pauline Kuss
Bio: I am a 3rd year PhD student at the Freie Universität Berlin with a research interest in agentic AI and sociotechnical approaches to AI governance.
Fellowship project: Governing AI Agents – An Affordance Perspective
Abstract: AI agents – AI systems that can autonomously plan and execute complex tasks with limited human oversight – offer significant opportunities but also pose critical challenges Including the controllability of their operations, liability allocation, and the prevention of harmful misuse. A better understanding of AI agents, what they are and how they realise their agenticness in real-world impact, is necessary to inform responsible agent development and effective governance responses. Contributing to this goal, this research explores the value of affordance theory to conceptualise AI agents in terms of the action potential they can perceive and realise within their environment. Affordances are an integral concept in sociotechnical research and its integration responds to recent calls to close the sociotechnical gap in AI safety research. Specifically, this research project 1) outlines the key characteristics and governance challenges of AI agents to motivate and inform regulatory action and further research. It then 2) conceptualises AI agents in terms of affordance realisation, and 3) provides a taxonomy of factors determining AI agent affordance realisation. Based on the foregoing, we finally 4) provide recommendations for agent design, environment design, and governance frameworks to reduce the risks and improve the management of AI agents. We also offer a list of open questions as guidance for further research. The research is mostly based on literature review. To refine our understanding of the challenges of AI agents and to improve the practical value of our results, we iterate our findings with experts from academia, AI labs, and governance. The project output will encompass a final research report addressed at AI labs and policy-makers. To invite early feedback, we furthermore plan two intermediary publications on the preliminary findings of step 1, and steps 2 and 3 described above. Suitable outlets tbd.
Theory of Change: Contribute to understanding how AI agents interact with their environment to create (harmful) effects > contribute to understanding how interventions in this AI-environment relation can be used to enable / constrain AI agents > inform AI agent / environment design and policy influencing this > improved controllability of AI agents > Reduce the risks of misuse and accidental harm by AI agents
Research interests: Agentic AI; sociotechnical AI governance; conceptualizing and theorizing agency; systems thinking
Contact: paulinekuss@gmail.com
Allison Huang
Bio: I’m currently an undergrad at University of Southern California studying an integrated degree in computer science, design, and business. I’ve previously done governance research on open-source models, and am hoping to get more technical experience with my ERA project.
Fellowship project: How well can LLMs defend against persuasive text in decision-making scenarios?
Abstract: LLMs have demonstrated the ability to generate persuasive text, with empirical studies suggesting that LLM-generated text is capable of influencing human decision-making. However, there is little formalized research on whether LLMs can be persuaded by the outputs of other LLMs, particularly in decision-making scenarios. This project seeks to fill that gap, and potentially address sub-questions including whether a less capable LLM can persuade a more capable LLM. At the core of our experimental setup, we define two LLMs: A and B. We will obtain a baseline/control by measuring model A’s behavior on an evaluation, then reevaluating A’s performance when model B persuades A to answer differently. We will use a decision-making evaluation, likely either in the domain of law or morality.
Theory of Change:
Research interests: evaluations, adversarial robustness
Contact: huangac@usc.edu
Isaac Robinson
Bio: My name is Isaac Robinson and I am a PhD student at Oxford University studying computer science. My degree focuses on algorithmic game theory and AI fairness, and I am hoping to do a deeper dive into frontier models, artificial intelligence, and governance over the course of my ERA project.
Fellowship project: The Rationality of AI Agents
Abstract: This research aims to measure the rationality of Large Language Models (LLMs) using game theory to determine if traditional economic theories and mechanisms can be used to control LLMs similarly to humans. We will transform classical game theory scenarios into narrative stories that obscure their game-theoretic origins, preventing LLMs from recognizing and relying on pre-learned patterns. We will start by selecting well-known game theory models, such as the Prisoner’s Dilemma and Stag Hunt, and convert them into detailed narratives. These stories will preserve the strategic choices of the original games without revealing their structure. LLMs, including models like GPT-4, will then interact with these narratives, making decisions at key points that correspond to game theory decision junctures. The LLMs' decisions will be analyzed to see if they align with rational strategies predicted by game theory, such as incentive compatibility. Metrics like the frequency of optimal strategy choices and decision consistency will be used to assess their rationality.The goal is to determine whether LLMs exhibit rational decision-making in these disguised scenarios and if economic theories and mechanisms can be applied to control their behavior. This study bridges game theory and AI, aiming to enhance our understanding of LLM decision-making and their potential for strategic applications in fields requiring sophisticated decision-making and control. Moreover, seeing how LLMs align with traditional economic theory could provide a framework for the utilization of mechanism design as a means to control super-intelligence agents.
Theory of Change: Modern economic theory relies on assumptions about how humans behave. As AI agents become an increasingly large portion of our economy, it is important to understand if they interact with these economic systems that were built to align the incentives of humans in a similar way.
Research interests: evaluations, adversarial robustness, mech interp, learning theory, game theory, law
Contact: isaac.robinson@cs.ox.ac.uk
Theory of Change:
Research interests:
Contact:
Tina Wünn
Bio: I have a BSc in Biology and an MSc in Medical Informatics, with a particular emphasis on AI applications in healthcare. Since graduating, I have contributed to two biosecurity policy research projects, and now want to focus on AI-biosecurity governance.
Fellowship project: Governance of AI-bio tools in the Global South
Abstract: The use of artificial intelligence (AI) in the life sciences offers considerable benefits, but it also presents risks of misuse. One growing concern is the potential of AI to facilitate the development of biological weapons. AI could reduce the barriers to creating these weapons, making them more accessible to a wider range of actors, and enabling the design of more harmful biological agents. In the Global South, efforts to address these risks have been limited. We will investigate actions policymakers can take to govern AI-bio tools in order to reduce the risks associated with bioweapons.
Theory of Change: AI-bio tools have the potential to make biological weapon development more accessible and increase their potential for harm. Currently, there is insufficient governance of these tools in the Global South. To address this, we will write a policy paper to provide policymakers with recommendations for implementing measures within their countries. Additionally, I will present a poster on this topic at the August 2024 Biological Weapons Convention (BWC) Meetings.
Research interests: AI-biosecurity, biological weapons, governance of emerging technology, Global South
Contact: t.l.wuenn@gmail.com
AI Governance
Alejandro Ortega
Bio: I’m about to start a 6-month placement at Apollo. I’ve been a freelance AI governance researcher over the past 9 months, looking into the history of nuclear power regulation in the US for Open Philanthropy, and voluntary safety in nuclear power and aviation for the Oxford Martin School. Previously I was EA Oxford Director and studied for an MSci in Physics and Philosophy at Bristol.
Fellowship project: Evaluating GPU self-destruct mechanisms for anti-tampering purposes
Abstract: In worlds where it is much harder for parties to secretly stop complying with the agreement, an international treaty on AI is i) more likely to be agreed in the first place and ii) more likely to be kept to. Hardware Enabled Mechanisms (HEMs) on GPUs can help verify that all parties are complying with their obligations e.g. on the maximum size of a training run, or what kind of workloads a GPU is being used for. But parties need to be able to trust that these HEMs are accurately reporting information order for these HEMs to build confidence, parties need to know that it’s difficult to tamper with them and have not been tampered with.
I’m looking into on-chip self-destruct mechanisms as a response to tampering, as a way of disincentivising actors from performing physically invasive attacks on GPUs (e.g. CCP performing attacks on GPUs on Chinese soil)
Theory of Change: Evaluate different self-destruct mechanisms-> influence future research by IAPS/Rand people about which routes are most promising + influence research direction of Davidad’s HEMs group + influence grantmaking in the area by Open Philanthropy
Research interests: GPU hardware, GPU firmware, infosecurity, banking/nuclear power/aviation/securities regulation, whistleblower protections, reporting requirements on deployment and cybersecurity incidents,, the Defence Production Act, embedded supervision, AI threat models, national security-relevant evaluations
Contact: alexor487@gmail.com, +44 7476951892
Madeline Proctor
Bio: I study Social Studies with a focus field in AI and the Law at Harvard, where I am writing a thesis on epistemological constructions surrounding digital trade goods within international law. My previous work includes assessing accuracy claims of LLM-powered legal technologies, plus lots of US constitutional law and policy.
Fellowship project: Mapping Political Considerations for a Strict Liability Regime with Expanded Punitive Damages for Advanced AI in the United States
Abstract: Technical uncertainty in AI safety poses immediate feasibility challenges to direct regulation. In addition to being more tractable, ex post tort liability regimes are a promising tool for deterring deployment of existentially-threatening AI systems due to passive scaling of liability in proportion to specific threats (Weil 2024). While the threat of litigation may disincentivize deployment of dangerous AI, the current US torts system is ill-suited to assign liability for harms from AI-produced existential risks. Previous work (Weil 2024) proposes two changes to the current regime to address these pitfalls : (1) Classifying advanced AI deployment as an abnormally dangerous activity warranting strict liability, and (2) Assigning punitive damages based on expected harms. Plausible legal arguments for high-risk algorithms falling under strict liability exist, but concerns on scope and degree of foreseeability remain. Assigning damages for expected harm would mean assigning liability for uninsurable risks generated, but not realized. Although this change may be necessary in internalizing catastrophic risks so large they are practically noncompensable, it would likely face greater opposition and require greater political will. This regime could be implemented using courts, where the main challenge is inconsistency with past doctrine. If implemented legislatively, though the lack of precedent is less directly relevant, a draconian understanding of the regime may emerge. This paper draws on theoretical considerations and expert interviews to assess feasibility of and identify key obstacles to implementing a regime like Weil’s in the United States. It maps potential normative, economic, and political costs and benefits to relevant stakeholders – including US policy teams, frontier model developers, and interest groups with public-facing stances on California’s SB-1047. This interest mapping aims to identify crucial cruxes and tipping points that can be used to streamline consensus-building around a regime change to counteract dangerous AI. Topics include foreseeability, allocation of liability for judgment-proof parties, innovation, and race dynamics.
Theory of Change: Contribute to gaps in US AI liability regime discussions, which may temporarily replace or work alongside other regulation to deter extreme risk. Build on normative calls for strict liability to understand political feasibility of similar legislation in the US. Stratify relevant stakeholder attitudes toward a torts regime which would greatly incentivize AI safety efforts.
Research interests: AI liability, philosophy of technology
Contact: maddieproctor@college.harvard.edu
Heramb Podar
Bio: Final-year student at IIT Roorkee, previous experience with the Center for AI and Digital Policy and the Millennium Project
Fellowship project: How to Harness a Llama: A Guide for Hosting Platforms to Secure the Open-Source Ecosystem
Abstract: Open-sourcing model components such as model weights, architecture, or code are considered to be a positive aspect of AI research and safety. However, the downsides of repurposing the code for misuse or use of the model weights to train other models with harmful applications or unexpected capabilities cannot be ignored. Further, the anonymous nature of users, the decentralized nature of network contributions and a lack of mandatory safety review of the AI models complicate the matter of mitigating proliferation or unexpected capabilities in the open-source domain.
We are targeting hosting platforms as they provide a centralized management system and a tangible chain of liability. These platforms have the power and responsibility to implement safety requirements, halt the distribution of harmful code, and ensure adherence to licensing and ethical guidelines.
This project will develop policy recommendations aimed at hosting platforms to implement methods that can reduce risks from open-source AI systems. Examples include code obfuscation, automated licensing compliance checks, and code signing. By introducing friction in the deployment and access processes at the point of proliferation, we aim to provide better-targeted defences now, thereby avoiding the need for broader, more restrictive regulations later, which could diminish the benefits of open-source development.
While the primary goal is to enhance safety, it is crucial to balance these measures against the benefits of open-source innovation. Our approach includes giving insight into the tradeoff of the adoption of such safety recommendations with potential risks to open-source development.
Theory of Change: Talking with software developers, studying policy paper, compliance requirements and coming up with policy recommendations to inform companies like GitHub and Hugging Face so that they take a proactive rather than reactive approach to prevent the proliferation or repurposing of code for misuse.
Research interests: AI standards, auditing, risk assessment, national AI policies
Contact: podar_hd@cy.iitr.ac.in or my LinkedIn
James Lester
Bio: I recently graduated from the University of Cambridge in Economics, where I also helped with outreach and community building for AI Governance and Effective Altruism. My undergraduate thesis modelled how expert advice on existential risk policy might break down under imperfect information and heterogeneous priors over risk levels. After ERA, I’m heading to Oxford to pursue the MPhil Economics.
Fellowship project: Assessing Risk Compensation Effects in the Deployment of Risky Technology - Evidence and Implications for AI Strategy and Policy
Abstract: Innovations which enhance the safety or reliability of AI systems might affect overall risk levels through two conflicting channels: (i) they reduce accident risk directly, and (ii) they might increase decision-makers’ willingness to deploy them, thus indirectly increasing risk. We aim to assess the prevalence, causes, nature and mitigation of such risk compensation effects in the deployment of AI. This work integrates (i) the existing theoretical literature addressing AI race dynamics, (ii) formal econometric work exploring risk compensation in individual behaviour, (iii) select insights from historical technological development and deployment, and (iv) the structural risk literature surrounding AI development and deployment. We then consider the relevance and potential implications of this analysis for the AI risk context, and thereby identify the most decision-relevant considerations around risk compensation for (i) higher-level AI strategy and (ii) individual research agendas in AI Safety. In particular, we develop a risk self-assessment sheet for interventions designed to promote AI safety, to aid structured reflection on the potential for risk compensation effects to arise.
Theory of Change:
Research interests: Global Priorities Research, Technological Progress and Diffusion, AI Threat Models, Formal Social Epistemology, Welfare Economics
Contact: james.lester@balliol.ox.ac.uk
Edward Kembery
Bio: MPhil in AI and Ethics from the Center for the Future of Intelligence, Cambridge; helped set up CDX, an AI advisory group for the Japanese Gov, and Cambridge AI Safety Hub’s policy wing. I wrote my thesis on mitigating the risks from emerging AI proliferation scenarios.
Fellowship question: What is the marginal net uplift in malicious actor capabilities from varying degrees of model access?
Abstract: Expanding the set of actors with access to key AI model components offers notable benefits, but could enable irresponsible or malicious uses of the technology. Whilst different strategies for structuring access to models have been proposed, it remains unclear how these different levels of access might augment malicious actor capabilities, making it difficult to calibrate structured access policies responsibly. This paper addresses this gap by evaluating the degree of net marginal uplift in malicious actor capabilities for each type of access proposed by Bucknall and Trager (2023): namely, sampling, fine-tuning, inspecting, modifying and observing metadata. The method will be a literature review and expert interviews.
Theory of change: The initial scope will be to evaluate just the net marginal uplift for actors aiming to perpetrate cyberattacks via training unsafe models, or by jailbreaking safe models. However, in doing so, the project aims to create an empirical framework which generalises across multiple threat vectors to inform responsible access policies, regulation standards, and future research, helping to inform a consensus around AI access. Notably, to better ensure robustness in the face of technological uncertainty, the paper describes how these risks might change depending on the raw model capabilities, its capabilities relative to the frontier, and the resilience of the broader AI ecosystem to adversarial attacks.
Research interests: Open-source, AI proliferation scenarios, long tail AI risks (neuro, drug, value drift), democratic inputs to AI governance
Contact: ejk64@cam.ac.uk
Theory of Change:
Research interests:
Contact:
Jai Patel
Bio: Recently completed MPhil Ethics of AI, Data and Algorithms @ Cambridge. Finished research on Responsible Scaling Policies. Experience in education and EdTech. Working @ UKAISI Safeguards Team. Remote ERA Fellow
Fellowship project: Developing an Agentic AI Crisis Response Framework (UK)
Abstract: Agentic AI systems, characterised by autonomous decision-making and adaptive learning challenge existing crisis response mechanisms. They might interact with critical infrastructure in complex, unforeseen ways, or adapt to containment efforts. The UK's current frameworks are likely insufficient to address the unique risks these systems present to national security and critical infrastructure. This study aims to develop a prototype crisis response framework for agentic AI incidents in the UK, incorporating relevant international insights for generalisability. The research will employ a mixed-methods approach, including a systematic review of UK and US AI legislation and institutions, semi-structured interviews with 8-11 experts, and scenario-based testing. The expected outcomes include a UK-specific typology of agentic AI risks, a prototype Agentic AI Crisis Response Framework, and a set of training scenarios. This research will fill a critical gap in UK national security preparedness and contribute to the global body of knowledge on AI safety and crisis management, potentially influencing policy and practice beyond the UK.
Alternative deliverables/ series of self contained outputs instead of a comprehensive report: threat assessment matrix, scenarios for tabletop exercises, policy memo for government stakeholder (leveraging UKAISI positioning for access to key stakeholders), conducting tactical decision making game with government/NatSec.
Theory of Change: Increased salience of agentic threats in national security community with concrete tools for mitigation → Help fill gaps in crisis response through scenario planning, conventions for disclosure and information sharing given the importance of rapid escalation procedures, containment, and mitigations → Make government/society more resilient to these threats as last line of defense if safeties fail.
Research interests: National Security, Responsible Scaling, Threat Modelling, Science of Evaluations, Democratisation of AI, Artificial Sentience, International AI Governance, Philosophy of AI
Contact: jai.patel12312@gmail.com
Damin Curtis
Bio: Master’s in International Affairs with a focus in technology policy. My previous research includes how advanced chip access impacts US security posture in East Asia, enforcement mechanisms for potential AI regulations, and the US AI governance institutional landscape.
Fellowship project: Coordination avenues between frontier AI developers and US defense/intelligence community
Abstract: Security threats from AI are manifold. However, if expertise on AI threats is not integrated into mainstream security establishment decision making processes, the US and its allies will likely under-invest in security mitigation strategies against AI-driven threats due to information deficit.
This paper evaluates the avenues by which the US defense/intelligence establishment becomes aware of security threats from AI model development/deployment. This paper provides an overview of current avenues and their barriers, presents proposals for improved coordination between AI developers and the US defense establishment, and identifies the main threat areas where coordination is especially important.
Theory of Change: identify institutional gaps where change could lead to increase information flow towards the relevant decision makers,, suggest ways to improve coordination channels to improve responsiveness and proper proactive assessment for security community to act on AI threats
Research interests: AI threat models, threat mitigation strategies, technology governance, chip access, US security domains
Contact: damincurtis@gmail.com
Duncan McClements
Bio: I’m an Economics student at Cambridge and a Research Associate at the Adam Smith Institute.
Fellowship project: Optimal punitive damages under mandatory insurance
Abstract: AI firms face differing incentives to those socially optimal. Strict liability is insufficient to align incentives given limited liability actors; while mandatory insurance remedies this for smaller scales due to insurers monitoring firms to reduce premiums, this remains insufficient if firms can cause damages greater than the value of any plausible insurers capital. This can be remedied by a punitive damage regime including expected unpriced damages - resulting in an escalating schedule of punitive damages, inferring firm risk choices from realised damages. We model the optimal schedule of such damages and estimate the corresponding reductions in catastrophic risk and welfare gains.
Theory of Change:
Research interests: Mechanism design,
Contact: duncanmcclements2@gmail.com
Ben Chancey
Bio: I study Philosophy and Computer Science at McGill, will probably want to go to law school, and will also probably pursue a career in AI Governance.
Fellowship project: Towards a Standard of Strict Liability for Harms Caused by Sufficiently Advanced AI Systems
Abstract: There are currently many regulatory proposals and efforts underway in the United States to address the risks posed by the emergence of increasingly powerful artificial intelligence (AI) systems. Regrettably, the state of the AI Alignment and Safety fields are such that even the highest safety standards and evaluations are not sufficient to protect against the risks this technology imposes. This is of particular concern as these systems’ capabilities scale toward existentially dangerous levels, a possibility now widely recognized by experts in the field. Accordingly, an alternative means of reducing the likelihood that potentially catastrophically harmful systems are deployed is needed. One such means could be to internalize their risks via an effective liability regime. Existing US liability doctrine, though, is not capable of doing this, one reason being that it would require plaintiffs to prove that AI companies were negligent in their design or deployment process. And because our understanding of AI alignment is generally poor, it is unlikely that most plaintiffs could demonstrate a reasonable alternative course of action by which a lab could have avoided causing the harm.This paper presents a solution to this problem: a legal argument, according to the Restatement Second and Third’s tests, for classifying the development of sufficiently powerful AI systems as abnormally dangerous activities. This classification would render actors strictly liable for serious harms resulting from its use, a liability standard which does not require negligence — only that the harm was foreseeable and proximately caused by the actor’s behaviour. It will also answer some normative and practical objections to the application of strict liability to even advanced AI systems.
Theory of Change: In producing strong arguments for classifying the development of AI systems as an abnormally dangerous activity, it would increase the likelihood that, in a lawsuit against it, an AI company could be held liable for harms according to a standard of strict liability. Because absent this standard, civil suits against labs are unlikely to be successful, this project would thus contribute to giving liability law the teeth it does not yet have, namely the power to disincentivize the development and deployment of potentially catastrophically harmful systems.
Research interests: AI Law, Policy, who knows what else
Contact: ben.chancey@mail.mcgill.ca