Navigating cybersecurity through the rise of AI security superintelligence

Joshua Saxe, co-founder, Abundant Security

We need to use policy to navigate society through the rise of superintelligent security agents

But today’s AI security policy discourse centers on anemic signals and controls

We should be using real world attacker and defender usage trends to understand the net expected harms of a model launch

Another problem is that today’s policy interventions imagine model launches as the main AI security risk object

But actually, cyber damages are smooth functions of ecosystem trends

Bending these trends in the right direction at the ecosystem level should actually be the main object of AI security policy

Additionally, the discourse biases towards AI security risk assessments, and not AI security benefits assessments

But AI may be our best opportunity to reverse longstanding harm trends due to the industrialization of fraud and ransomware

My thesis

• We urgently need a large, capable AI cybersecurity observatory
• It would focus on
• … reducing net harms at the ecosystem level
• … maximizing AI’s defender benefits
• … and mitigating its security risks
• It would track the full breadth of AI security ecosystem and model capability signals
• It would recommend a full menu of policy actions

My thesis: today vs. the AI security observatory we need

My fantasy of what the AI security observatory would do

A prototype of the contrasting signals an AI security observatory would develop

Misalignment and reward hacking are real; coding agent damages are scaling exponentially.

But such incidents contrast with human-error damages which AI could potentially help avoid.

AI-mediated phishing is occurring at scale.

But AI’s use in phishing should be weighted against its ubiquity in phishing defense.

We now have clear real-world examples of superhuman agentic hacking

These incidents contrast with success in burning down security tech debt.

Overall, the current security crisis predates AI; there’s an opportunity for AI to provide net benefit here, with the right policies.

A prototype of the policy controls a security observatory would help us navigate

Using subsidies and regulation to incentive critical infrastructure hardening

Regulations and support around inference provider know your customer programs

Subsidies and government support for AI cyber defense adoption

Phased rollout of models where it makes sense, accelerated rollout where that makes sense

Building an AI security observatory is urgent; we may soon be living in a very different world

Dual-use cyber capabilities are on an exponential; the stakes keep rising

Dual-use capabilities and attacker access keep getting cheaper, adding urgency

The loss landscape will be complex and increasingly spiky due to AI’s dual use nature; policy precision is key

Thanks; space for questions and discussion�josh@abundantsecurity.co�@joshuasaxe on X