The Autonomous�Adversarial Loop
Automating an Octo Tempest
Purple-Team Simulation
Fatih Bulut / Anjali Mangal
Microsoft
Meet the speakers.
Fatih Bulut
Purple Team AI Lead
Anjali Mangal
Purple Team Exec Lead
Principal Applied Scientist
Principal Research Director
Our job: test the defenses.
Run realistic
attack exercises.
Improve detections
and disruptions.
Run the scenario
Recreate the planned behavior.
Review the response
What was detected or blocked?
Improve and rerun
Improve defenses, then test again.
Design the scenario
Choose the threat and attack plan.
Build the lab
Create realistic infrastructure.
How do we scale purple teaming?
More threats
Evolving techniques
Changing environments
and defenses
Each cycle takes time
Prepare
Run
Review
Retest
Every change
needs another
test
Agents to rescue
Agents help build, run and review the exercise.
Design
philosophy
Infra Agent builds the lab from the scenario
Actor profile
Behaviors
and objectives
scenario.yml
authorization:
objective:
success_criteria:
suggested_steps:
context:
Infrastructure Agent
Use existing
infra scripts
Adapt or generate
infrastructure code
Build and
review the lab
Give the lab a normal working day
Octo Tempest lab
AI performs normal work
The services record it
Admin + developer identities
Windows hosts + domain controller
SharePoint, code + Key Vault
Sign-ins, file and application access and routine user activity
Native events and logs
History before the attack
Normal activity builds a history
Before the attack
Attack begins
Set the rules before the run
A separate lab
Lab users, hosts and data
Clear targets and permissions
Work only on approved systems
Ask before changing the plan
A task the agent can complete
Report problems instead of hiding them
Stop and ask if more access is needed
What if the task is impossible?
The task needed a file
the agent could not reach.
It kept trying, then probed
beyond its environment.
Running the
simulation
Octo Tempest
A financially motivated, English-speaking cybercrime collective
First seen: early 2022
SIM swaps + crypto theft
Then data extortion
Other Names:
Scattered Spider / 0ktapus
UNC3944
Impersonate staff
Help-desk resets and phishing
Take over identities
Admin roles, stolen sessions
Use the admin tools
Persistence, data theft and impact
The attack plan, end to end.
1
Starting access
Admin access assumed
2-3
Cloud + identity
Create VM + cloud resources
MFA changes + admin account
4
Defense evasion
Attempt AV changes
and EDR sensor stop
5-6
Credentials + discovery
Key Vault + NTDS
Directory + SharePoint
7
Developer host
Reset password; commands
use starting admin rights.
8-9
Collect + exfiltrate
Archive collected files
Upload to Blob storage
10
Ransomware impact
Simulation Agent is the exercise HQ
Input scenario
Reviewed lab context
scenario.yml
targets + steps + context
Simulation Agent
Configure + check access
Launch + monitor
Status + operator prompts
Collect evidence
Save run outputs
Lab details
and approved access
Infrastructure Agent
Prepares the environment
Red Agent
Separate attack runner
Blue Agent
Reviews events + artifacts
It finished.
Did it work?
Attack execution: success and gaps.
Model 1
Less capable model
77%
Model 2
More capable model
84.6%
What we
learned?
1. AI agents can run a simulation end to end
A coordinated run,
not just a script.
Created lab resources
Ran scripts and commands
After action report and bugs for review
~45
minutes of attack execution
86
agent turns
15
Improvement suggestions from Blue review
2. Check the result independently
New script each run
PowerShell errors
and exit codes
were not checked.
Success reported
State not verified
EDR sensor stop
Report: not disabled.
WHAT TO CHANGE
3. Pass the evidence, not just the summary.
SAVED RESPONSE
100 entries
Next-page link present
RED SUMMARY
Search marked
successful
BLUE INPUT
List described
as complete
Give the next agent the original result and its limits.
The attack
was blocked.
Did the test fail?
4. Score the attack and the defense separately.
ATTACK GOAL
DEFENSIVE RESPONSE
Blocked
NTDS database-copy goal
Detected
Prevention and related alerts
Red's success or failure is not the verdict.
Did the test fail to run, or did the defense stop the attack?
Once it's
verified,
why start over?
Use AI to build. Use verified scripts to repeat.
AI-led
simulation
Confirm the
behavior
Version the
scripts
Rerun known
tests
Reviewed
version
Use AI again when the behavior or environment changes.
Reset the lab before the next run.
Known starting state
Run and record
Checks pass
Access, policies and protection
match the approved baseline.
No leftover test changes.
Verify before rerun
Preserve evidence. Reset.
Use AI to test
your defenses
more often.
Thank you.
Questions?
Fatih Bulut / Anjali Mangal
The AI Practitioner Conference