Human augmentation by AI agents in C2:

Piercing the fog of war and detecting strategy using StarCraft II for C2

Presenter:

Carolina Sanchez

Senior AI Assurance Consultant

OFFICIAL

Introduction

​

The research reported on in this paper was funded by the UK MOD Machine Speed Command and Control (MSC2) project. This project was part of the UK Defence Science and Technology Laboratory's (Dstl) AI Programme with the intent to transform C2 by enabling more `timely and effective C2 processes across all environments, domains and levels of command, so the Defence enterprise can anticipate and adapt more successfully than adversaries.

​

This paper is one of six presented at the 29th ICCRTS which document different aspects of the MSC2 project which explored the feasibility of a Human Agent Collective (HAC) that combines human insight with machine speed AI agents employing shared digital artefacts, shifting C2 from human teams to human-machine teams, where humans and AI work together.

​

​

2 of 13

OFFICIAL

OFFICIAL

Executive summary:��Human augmentation by AI agents in C2:�Piercing the fog of war and detecting �strategy using StarCraft II�

This opportunity explores the acceleration of military research by using gaming centric AI as another type of simulation environment, to test and validate approaches at speed and transform these learnings into applications to military use cases.

​

These AI agents can form part of a future Human Agent Collective (HAC).

​

Our achievements advance research and knowledge on Human augmentation by AI agents in C2:

​

3 of 13

OFFICIAL

OFFICIAL

�Situation: AI research in video games provides real value for MOD research

Gaming provides a testbed for elements of future C2 and MOD concepts. Games are simplified models of a reality, and by addressing problems in these environments we can learn how to solve analogous problems in real applications faster. Gaming provides value for MOD AI research because:

​

• Realistic military problems present critical data challenges. Gaming provides a “surrogate task” to explore these.
• AI research in the gaming space can be leveraged to accelerate AI R&D
• AI research in gaming is a testbed for understanding Human-Machine interactions

​

​

4 of 13

OFFICIAL

OFFICIAL

Blue force challenges:

• Fog of war/Partial information
• Unknown opponent strategy

​

AI information provision:

• Piercing the Fog of war
• Opponent strategy
• Confidence levels and explanations

Human Augmented decision making

AI and Explainability framework

StarCraft II

User centric AI delivery

1

2

4

5

Usability study

AI functionality development

3

How to understand to the user and meet user requirements and needs for functionality, explainability and interactions

​

5 of 13

OFFICIAL

OFFICIAL

StarCraft II for AI Research

Video games are a valuable environment for AI research and demonstration because they provide:

​

• A robust simulation environment within which AI models can be trained and tested without the expense of collecting real-world data or the risks associated with operating AI agents in real-world scenarios.
• Large datasets of past games played by human players, providing training and testing data and often showcasing high performing strategies developed by professional human players.
• Clear measures of success inherent in the competitive design of many games which allows games to serve as effective benchmarks.
• A significant amount of AI research has made use of the game StarCraft II (including AlphaStar by DeepMind that has evolved into AlphaFold (protein folding).
• Fog of war and rapid decision making makes this game extremely useful to explore the challenges of decision making with partial information.

​

StarCraft II

1

Battle.net® ©

1996 - 2002 Blizzard Entertainment, Inc.

All rights reserved

6 of 13

OFFICIAL

OFFICIAL

User-centric Approach

We put the user at the centre of our approach to development and created a user centric Explainability framework

to understand the users' needs for AI functionality but also for understanding and effective use of the AI outcome.

AI and Explainability framework

2

7 of 13

OFFICIAL

OFFICIAL

Approach: AI decision support in StarCraft II for red force location and inferring strategy

Blizzard Entertainment starcraft2.com

Key learnings:

​

1. AI opponent nowcasting location is possible with only partial information
2. AI works better for predicting some units than others. This can inform ISR to direct resources towards those that the AI is worse at and use the ones AI is good at

​

Key learnings:

​

1. We reduced the amount of data needed to extract relevant information (key features)
2. As strategic features importance changes over time, AI helps to detect non- directly observed changes in opponent strategies

AI functionality development

3

8 of 13

OFFICIAL

OFFICIAL

Results: Our Human-centric approach and AI explainability augments intelligence on opponent location and strategy for operational and tactical decision-making

Our human-centric approach was translated into a UX interface where the results of the AI agents were presented graphically and as textual summaries:

1. Information on location of enemy units through fog of war
2. Opponent strategy prediction and changes over a timeline
3. Human understanding the AI output with graphical and textual explanations

To validate this work, we performed a usability study to assess what users thought of how the information was displayed for their decision making.

​

User centric AI delivery

4

9 of 13

OFFICIAL

OFFICIAL

Internal Usability study

Human Augmented decision making

5

Usability study

Study protocol

Our internal usability study recruited internal employees in order to get their opinion on the use of explanations and the display of these explanations.

50% of them considered themselves intermediate, 20% advanced and 10% experts.

10 of 13

OFFICIAL

OFFICIAL

Internal Usability study – Outcomes

The main takeaways were:

• Reliability: The graphical UI consistently had ratings above 3 (out of 5) for each video segment. The control UI had started off with similar reliability ratings to the graphical UI, but scores decreased steadily as each time segment went on, e.g. to an average rating of 2.1 for time segment 4 which is quite advanced in the game.
• Efficiency: The graphical UI received the highest average scores for all time segments. However, in general, none of the UIs were considered efficient for playing SC2, understandably due to lack of integration with the game.
• Trust (overall): The control UI was rated as being the least trustworthy UI. Both the graph and the text UI were rated as more trusted, with the graphical UI having ratings above 3 throughout.

​

Human Augmented decision making

5

Usability study

11 of 13

OFFICIAL

OFFICIAL

What comes next?

12 of 13

OFFICIAL

OFFICIAL

^{© Cambridge Consultants 2024}

UK

Cambridge Consultants is part of Capgemini Invent, the innovation, consulting and transformation brand of the Capgemini Group

www.cambridgeconsultants.com

USA

SINGAPORE

JAPAN

.

.

.

31 May 2024

P5211-P-021 v0.3

OFFICIAL

OFFICIAL