How to securely deploy agents that make sensitive decisions autonomously

Joshua Saxe

AI security engineering @ Meta

Agent security doesn’t reduce to traditional security, or traditional AI alignment

AI security does not reduce to the alignment problem

The alignment problem

There’s a philosophical ceiling on our ability to solve the alignment problem that’s too low

Fortunately, in security, we’ve spent decades building controls that assume misalignment, malice, and human error

But AI agent security doesn’t reduce to traditional security either

• We’ve never had to secure objects anything like AI agents before
• Agents can’t separate data from instructions and will always have prompt injection risk
• They don’t afford traditional program analysis
• You can read AI agents’ minds, pause them, roll them back, to enforce security
• You can ethically shrink and expand an AI agent’s freedom
• We can force them to think for an hour before making a sensitive decision

The thesis of this talk

is that AI agent security requires a novel fusion of cybersecurity, which assumes misalignment, and alignment, which maximizes p(aligned)

Real life photos of maximizing p(alignment) while integrating security

AI agent

2. Agent operates within ‘trust zone’ granting least necessary privilege; fine tuning and guardrails provide ‘best effort’ protection within this zone

1. AI agent we align through fine tuning, scaffolding, and guardrails

3. Security controls like identity, access management, and taint analysis, guarantee agent has no freedom outside of ‘trust zone’

Coarse model for how alignment and cybersecurity work together here

We need to manage dialing up the size of the trusted region over time with the help of red teaming, evals, and real-world experience

Security is the bigger, older discipline; what are the security lessons for agent security?

We built PCs to run arbitrary code from misaligned humans and then incrementally patched the problem with imperfect guardrails

We built the Internet on memory unsafe code and then incrementally patched the problem with partial defenses

Too often the barrier between the untrusted world and our most trusted assets was a series of probabilistic coin flips

Of course there’s no risk we’re doing this today

Of course there’s no risk we’re doing this today

Lessons for AI agent security

• We should have built platforms that limited the trust radius around known catastrophic risks
• We should never have oversold probabilistic defenses as fix-alls
• We should have used probabilistic defenses to mitigate intentionally accepted, residual risks
• We are traumatized by having to solve generations of ill-posed problems, but we need to heal and demand better this time

AI agent

2. Agent operates in a trusted region in which we allow it freedom of operation

1. AI agent we align through fine tuning, scaffolding, and guardrails

We need to execute well on building this

3. Security ensures its actions never affect this untrusted region

And we’ll need to elegantly manage the risks in dialing up trust

On what we will need to build

The linkages between alignment and security components are shown on the right

Agents need to integrate with traditional security systems in tool calling auth, identity systems, PKI, etc

How the new security stack will fuse AI- and security-native concepts

Guardrails should use ML methods but also security tools like static and taint analyzers

How the new security stack will fuse AI- and security-native concepts

Deterministic controls should use context window data provenance as part of decisions

How the new security stack will fuse AI- and security-native concepts

Detection and response operations should add understanding of objects like chains of thoughts, context windows, and instruction hierarchies

How the new security stack will fuse AI- and security-native concepts

Fusing elements of machine learning safety and alignment and security is a long-term project

Work that steps us towards fusing security with AI alignment

”The Camel paper” (Debenedetti et al., Google Deepmind))

https://arxiv.org/pdf/2503.18813

Work around security infrastructure for AI agents

Work around developing chains of thought as useful security objects

Work around applying static code analysis tools to LLM code outputs

Wrapping up

The thesis of this talk

is that AI agent security requires a novel fusion of cybersecurity, which assumes misalignment, and alignment, which maximizes p(aligned)

We need to manage dialing up the size of the trusted region over time with the help of red teaming, evals, and real-world experience

AI agent security will be a rollercoaster, but this time can be different

• We have 30+ years of experience building security amidst technical paradigm shifts
• We have 10+ years of work around adversarial ML and AI alignment
• We have robust policy engagement and willingness to build security capacity and standards cross-industry
• Let’s fuse the best of security, alignment, and the lessons of the past, to create a better security model this time