1 of 30

Risks of AI as Economic Agents

Jonathan Stray

UC Berkeley CHAI

2025-3-19

2 of 30

Business Integration is Coming

​

3 of 30

Anthropic’s MCP allows LLMs to control apps

​

Excerpt from list of current MCP servers, 2025

4 of 30

Misalignment

​

​

​

5 of 30

OpenAI, 2016

The King Midas Curse of Simple Reward Functions

​

6 of 30

Eventually our society’s trajectory will be determined by powerful optimization with easily-measurable goals rather than by human intentions about the future.

​

We will try to harness this power by constructing proxies for what we care about, but over time those proxies will come apart.

​

Corporations will deliver value to consumers as measured by profit. Eventually this mostly means manipulating consumers, capturing regulators, extortion and theft.

Profit is a Misaligned Objective

What Failure Looks like, Christiano 2019

“

7 of 30

Slop

​

​

​

8 of 30

Here Lies the Internet, Murdered by AI, Erik Hoel 2024

AI-Generated “Slop”

​

9 of 30

AI-Generated “Slop”

​

10 of 30

Content Farms, circa 2010

​

11 of 30

Closed-loop Optimization for Audience is Coming�

​

Unless appropriately designed and supervised, results will include:

  • Crime, violence, car crashes, gore, realistic depictions of death
  • Pornography, including celebrities and children
  • Conspiracy theories, misinformation, extremist politics
  • Extremely strange children’s entertainment

12 of 30

Fraud

​

​

​

13 of 30

Several Prominent Twitter Accounts Hijacked in Cryptocurrency Scam,

Bank Info Security, 2020

Human Crypto Fraud

​

14 of 30

Pallisade AI Research, 2025

LLMs will cheat to win at chess

​

15 of 30

Detecting misbehavior in frontier reasoning models, OpenAI 2025

LLMs will disable their test code

​

16 of 30

Palisade Research, 2024

AI Hacking Skills Rapidly Improving

​

17 of 30

Detecting misbehavior in frontier reasoning models, OpenAI 2025

AI Attempts to Hide Bad Behavior

​

18 of 30

Profit Maximizing AIs Will Commit Fraud and Hide It�

​

Unless appropriately designed and supervised, results will include:

  • Reinvention of classic scams, including inferior or non-existent products
  • Illegal trading in drugs, weapons, tax evasion, etc.
  • Externalities e.g. environmental destruction, privacy violations, discrimination etc.
  • Hacking for theft, espionage, promotion,etc.
  • If given legal knowledge, will exploit loopholes and weak enforcement
  • If penalized for bad behavior, will learn to conceal it from humans

19 of 30

Financial Instability

​

​

​

20 of 30

Each day, bot A sets its price to 90% of B, to win the sale by undercutting.

Each day, bot B sets its price to 130% of A, and sources the book from A if it makes a sale.

Individually Sensible Strategies can be Jointly Unstable

21 of 30

Jay Taylor, 2015

Algorithmically Generated “Flash Crashes”

22 of 30

Existing Protections Against Systemic Instability Are Weak�

​

Unless appropriately designed supervised, results will include:

  • Agents will adopt profitable but jointly unstable trading strategies
  • Bubbles, predominance of pump-and-dump strategies
  • Fast, silent exploitation of loopholes and edge cases not foreseen by regulators
  • Exploitation of weaker market participants e.g. frontrunning

​

23 of 30

Collusion

​

​

​

24 of 30

25 of 30

We find that market-level margins do not change when only one of the two stations adopts, but increase by 28% in markets where both do. These results suggest that AI adoption has a significant effect on competition.

Algorithmic Collusion in German Gasoline Prices

Algorithmic Pricing and Competition: Empirical Evidence from the German Retail Gasoline Market

Assad et al. 2020

“

26 of 30

The complaint alleges that RealPage contracts with competing landlords who agree to share with RealPage nonpublic, competitively sensitive information about their apartment rental rates and other lease terms to train and run RealPage’s algorithmic pricing software. �…

in a free market, these landlords would otherwise be competing independently to attract renters based on pricing, discounts, concessions, lease terms, and other dimensions of apartment leasing.

DOJ Files First Algorithmic Pricing Suit

Justice Department Sues RealPage for Algorithmic Pricing Scheme that Harms Millions of American Renters

justice.gov, 2024

“

27 of 30

Antitrust lawyers have raised concerns that if AI algorithms were able to control pricing decisions, were programmed to maximize profits, and had access to public information about competitors’ prices, then competing firms might arrive at a collusive outcome without any communication. Indeed, since AI agents aim to maximize the profits of individual firms, there may be no violation of price-fixing or antitrust collusion laws.

Current Legal Regime May Be Insufficient

Price collusion using artificial intelligence

Charles River Associates, 2023

“

28 of 30

We Do Not Know How To Prevent AI Collusion�

​

Unless appropriately designed supervised, results will include:

  • Every market becomes a cartel
  • AIs do not even need to communicate secretly if they are able to monitor rivals’ public prices.
  • Will we even know?

​

29 of 30

Has There Ever Been

A Truly Profit Maximizing Corporation?

30 of 30

The World that Profit-Maximizing Agents Will Create�

​

Illegal activity

Scams, fraud, drugs, weapons, hacking

Gray Areas

Machines will continually test the limits of law and enforcement.

Externalities

Environmental destruction and human rights violations

Market Instability

Systemic stability is a classic public good

Algorithmic Collusion

Every market AI enters will become a cartel