1 of 18

Assessing Distributed Sensemaking Performance in Command and Control: Results from an Exploratory Study

Emmeline Elliott and Simon Attfield

(Leggatt, A., Baber, C., Attfield, S., Minocha, S., Elliott, E., Fields, B., Hutton, R., Cowell-Smith, P., McNeillie, S.)

22 October 2022

2 of 18

Overview of Research
Nine Principles of Distributed Sensemaking
Overview of the Exploratory Study
Findings
Next Steps

Presentation Overview

OFFICIAL

3 of 18

Issue

OFFICIAL

The Future Operating Environment (FOE) is likely to be one in which operational combat units are required to be increasingly mobile and geographically dispersed with more decentralised Command and Control (C2) structures. Dispersed Military Teams:

Have access to different information
Sometimes without access to a higher commander
Leading to a greater emphasis on local and collaborative sensemaking capabilities to interpret and assess complex situations and to adapt and act effectively.

4 of 18

Solution

OFFICIAL

Sensemaking

Sensemaking is “A motivated, continuous effort to understand connections (which can be among people, places, and events) in order to anticipate their trajectories and act effectively” (Klein et al., 2006.)
Distributed sensemaking is the process by which people:

“seek data and apply their values and goals, expertise and domain knowledge to synthesise data, draw inferences and share situation-pictures with other military units for collective understanding and co-ordinated action” (Elliott et al., 2020.)

Potential Benefits for Dispersed Forces

Increase the speed with which distributed groups achieve a common understanding
Increase the quality and speed of decision making
Reduce their vulnerability to loss of communications through a better understanding of shared intent and increasing their efficiency in recovering shared understanding when communications are returned

5 of 18

Provide sufficient cues for sufficient sensemaking
Support low-cost information workflows *
Represent information quality and provenance
Promote expertise/domain knowledge
Allow time to acquire data/information to build an evidence-based and coordinated situation picture
Use strategies for the negotiation of sense *
Where appropriate, use strategies for frame enumeration and elimination
Provide explanatory context for actions, orders and requests *
Minimise the costs of achieving and maintaining common ground

For a detailed explanation of these principles please refer Elliott et al. (2020)

Defining Principles to Support Distributed Sensemaking

OFFICIAL

6 of 18

Research Question 1 (RQ1): How do distributed groups negotiate DSM problems and what factors affect this?
Research Question 2 (RQ2): How does an elaborated reporting format design impact upon DSM?
Research Question 3 (RQ3): What are the measurement issues associated with DSM?

-- An exploratory study – not hypothesis testing --

Research Questions

OFFICIAL

RQ1 was explored by imposing different challenges on the teams in the form of communications challenges (participants communicated via normal voice/radio communications in one condition, but primarily via text chat in another condition) and it was observed how experienced military participants go about distributed sensemaking when presented with a sensemaking challenge.

RQ2 was specifically driven by the elaborated reporting format versus a more conventional reporting format. The elaborated reporting format is intended to reflect an underpinning description of sensemaking as “abductive reasoning”. The underpinning logic of abductive reasoning in the military context provides a theoretically grounded frame for our research questions but most importantly for the logic by which we manipulated information in the experimental design through the briefings that were provided to the team participants (seeding aspects of “prior knowledge”) and the presentation of cues and information across the communications network based on sightings, reports and updated intelligence reports.

DSM performance was be measured by analysing the conclusions drawn by the participants from the communications traffic. We also analysed their choices and decisions in the dilemmas presented in the various scenario runs.

RQ3 was assessed by the ability of the various measures and analysis applied to the data collection effort to distinguish between different levels of DSM performance as well as the different strategies and techniques applied by the teams to conduct their DSM and whether their behaviour was consistent with the DSM principles identified previously.

7 of 18

Participants:15 serving Army personnel - 14 officers (Captain or Major), 1 warrant officer.
Participants worked as teams of 3 Company Commanders with comms (Battle Group Net and Company Nets).
Battle Group Commander not present.
Excon feeding each Company Commander their own unique reports.
4 task runs (35min), counterbalanced for order.

Study design

OFFICIAL

	Reporting Format	Run 1	Run 2	Run 3	Run 4
Team A Day 1	Conventional	Voice	Voice	Text	Text
Team B Day 2	Conventional	Text	Text	Voice	Voice
Team C Day 3	Elaborated	Voice	Voice	Text	Text
Team D Day 4	Elaborated	Text	Text	Voice	Voice

8 of 18

Scenario

OFFICIAL

9 of 18

Procedure & data capture

Stage	Activity	Data	Details
Pre-study		Biographical data	e.g. experience of company command, experience of working together
Study run (x 4)	Task (35 min)	comms observation	BG Net, Company Net – transcribed (semi-automated analysis e.g. topic, turns, social network analysis e.g. network density) physical interactions with map etc.
	Individual sensemaking questionnaire (ISMQ)*	responses	self-reported level of sensemaking
	Peer-to-peer discussion (10 min)	comms	transcribed
	Individual sensemaking questionnaire (ISMQ)*	questionnaire responses	self-reported level of sensemaking
Post-study	Individual interview		transcribed
	Group debrief interview		transcribed

compare

* Alsufiani, K (2020) Measuring the Effect of User-Generated External Representations during Sensemaking in Electronic Environments, Doctorial dissertation, Middlesex University, London, UK.

10 of 18

Conventional vs Elaborated Reporting (between participants)

Conventional reporting - conveys facts with no inferencing [A. Enemy; B. Own Situation; C. Admin; D. General]
Elaborated reporting – additional conclusion [E. Assessment / Intentions]

Voice vs text (within participants)

Voice+Text: low risk C2 State/low level Emissions Control (EMCON) condition
Text-only: minimal electronic emissions or low signature or bandwidth requirements

Manipulations

OFFICIAL

11 of 18

Example Findings

OFFICIAL

ISQM scores before and after peer-to-peer discussion, by voice and text.

Error bars represent 95% confidence intervals.

12 of 18

Example Findings

OFFICIAL

Peer-to-peer discussion time for conventional vs elaborated reporting formats.

Error bars represent 95% confidence intervals.

13 of 18

Comparing voice and text on properties drawn from comparative

assessments made by participants during interviews

OFFICIAL

PROPERTY	VOICE	TEXT
Feel/tone (Fingerspitzengefuh)	Good – Toneful. Easy to gauge emphasis, urgency.	Poor – Toneless. Hard to gauge emphasis, urgency.
Interpersonal interactivity	Good – Good for direction.	Poor – Good for information.
Attentional selectivity	Good – Easier to tune in/out	Poor – Harder to tune in/out
Sensemaking ‘spotlight’	Broad – Draws user to a broader situation picture.	Narrow – Draws user to a narrower situation picture.
Interference with visual channel (environment, map, weapon system).	Low – Uses auditory channel.	High – Uses visual channel.
Information creation effort	Low – Talking has low(er) cost. Easier to add value.	High – Typing has high(er) cost. Harder to add value.
Reviewability (Clark and Brennan, 1991)	Poor – Non-persistent. Detail can be lost. Often accompanied by compensatory note-taking. Hope you capture the right thing.	Good – Persistent. Detail retained. Review by scrolling. Could be enhanced with search function.
Impact of ambient noise	High – Uses auditory channel.	Low – Uses visual channel.
Information transfer effort	High – No copy and paste.	Low – Use copy and paste.
Suited for…	Direction	Information

14 of 18

Groups improved their ability to negotiate sense during the study.
Peer-to-peer discussion improved sensemaking (ISMQ measure and participant feedback).
Voice more efficient in creating a common frame for sensemaking compared to text.
Prior experience of company command led to higher self-reported sensemaking (ISMQ).
Prior experience of working together led to higher self-reported sensemaking (ISMQ) with less benefit from peer-to-peer discussion.

Research Question 1: How do distributed groups negotiate DSM

problems and what factors affect this?

OFFICIAL

15 of 18

Elaborated reporting format resulted in reduced peer-to-peer chat time.
Shorter messages were used - implying different sensemaking strategies.
Improved ISMQ scores.
Largely not visible to participants.
Elaborated reporting was about exposing reasoning and suggests the value of studies exploring this.
Report format influenced the way information was communicated, impacting message length measured both in terms of time and words.

Research Question 2: How does an elaborated reporting format

design impact upon DSM?

OFFICIAL

16 of 18

The DSMIQ was quite time intensive to administer.
The ISMQ appeared to be a sensitive measure of individual sensemaking.
Analysis of patterns of communication (network density, message length etc.):

Could be indicative of effective sensemaking.
Transforming raw data for automated data analysis takes time and effort.
Through analysing patterns of communication it could be possible to determine when and how Frames are introduced, elaborated, questioned.

Research Question 3: What are the measurement issues

associated with DSM?

OFFICIAL

17 of 18

Exploratory study successfully allowed the investigation of the three guiding research questions
Identified a number of variables (message elaboration, message medium, peer-to-peer chats, knowledge of other team members and previous command experience) that impact on DSM performance
Further research is recommended with a focus on investigating the principles further and:

Improving the measurement of DSM, including potentially automated communication analysis
Reducing the number of items in the ISMQ
Testing surrogate measures of DSM such as the time needed to conduct peer to peer sensemaking chats

Conclusion

OFFICIAL

1 of 18

2 of 18

3 of 18

4 of 18

5 of 18

6 of 18

7 of 18

8 of 18

9 of 18

10 of 18

11 of 18

12 of 18

13 of 18

14 of 18

15 of 18

16 of 18

17 of 18

18 of 18