Assessing Distributed Sensemaking Performance in Command and Control: An Exploratory Study

Emmeline Elliott, Simon Attfield, Andrew Leggatt and Chris Baber

(Leggatt, A., Baber, C., Attfield, S., Minocha, S., Elliott, E., Fields, B., Hutton, R., Cowell-Smith, P., Harryman, J.)

​

OFFICIAL / © Crown copyright 2020 Dstl

13 October 2021

• Overview of Research
• Nine Principles of Distributed Sensemaking
• Overview of the Exploratory Study
• Next Steps

​

Presentation Overview

OFFICIAL

Issue

OFFICIAL

Dispersed Military Teams

• Work in separate locations with access to different information
• Sometimes without access to a higher commander to give over sight of the situation

Issue

• More difficult to achieve a shared understanding of the situation because they have differential access to information
• Differences in interpretation due to individual differences in experience, skills
• Leading to conflicting orders and an inconsistent response to events

​

Technology Issue

• More reliant on technology to support communications
• More vulnerable to disruption from electronic counter measures and cyber threats

​

UK MOD © Crown copyright 2021

Solution

OFFICIAL

Sensemaking

• Sensemaking is “A motivated, continuous effort to understand connections (which can be among people, places, and events) in order to anticipate their trajectories and act effectively” (Klein et al., 2006.)
• Distributed sensemaking is the process by which people:

“seek data and apply their values and goals, expertise and domain knowledge to synthesise data, draw inferences and share situation-pictures with other military units for collective understanding and co-ordinated action” (Elliott et al., 2020.)

Potential Benefits for Dispersed Forces

• Increase the speed with which distributed groups achieve a common understanding
• Increase the quality and speed of decision making
• Reduce their vulnerability to loss of communications through a better understanding of shared intent and increasing their efficiency in recovering shared understanding when communications are returned

Suppliers: Trimetis Ltd, University of Middlesex, University of Birmingham and MASS Consultants Ltd

Research Tasks

• Dstl tasked the suppliers with the following:
• Task 1: Produce a definition, principles and a measure of distributed sensemaking and suggest an intervention to improve distributed sensemaking.
• Task 2: Experimental research to explore the principles of distributed sensemaking and associated interventions.

​

Research Task

OFFICIAL

Klein’s Data/Frame theory

Overview of Sensemaking

OFFICIAL

The Data-Frame model of Klein (replicated from Klein et al., 2006.)

1. Provide sufficient cues for sufficient sensemaking
2. Support low cost information workflows
3. Represent information quality and provenance
4. Promote expertise/domain knowledge
5. Allow time to acquire data/information to build an evidence-based and coordinated situation picture
6. Use strategies for the negotiation of sense
7. Where appropriate, use strategies for frame enumeration and elimination
8. Provide explanatory context for actions, orders and requests
9. Minimise the costs of achieving and maintaining common ground

Task 1: Defining Principles to Support Distributed Sensemaking

OFFICIAL

1. Provide sufficient cues for sufficient sensemaking

Summary: The dynamic of sensemaking is to alternate between data and developing a frame where the frame offers a plausible explanation of the data (Klein et al., 2007). It has been described as the enlargement of small cues. The cue that alerts a sensemaker to a situation, however, is rarely definitive and can lead to additional information seeking to test uncertain frames or to disambiguate competing frames (e.g. diagnosis) (Attfield et al., 2011). Further, the range of critical situations can be relatively small, but it can be important to disambiguate them quickly to inform action. Depending on their roles, values and goals, distributed sensemaking units operate more efficiently and more effectively through the prior anticipation of cues or cue-patterns relating to significant phenomena, and using this to inform the training of personnel and artificial intelligence systems and capabilities for sensing, recording and displaying data.

​

Enhancing Sensemaking

Distributed Sensemaking and the Future Operating Environment

OFFICIAL

6. Use strategies for the negotiation of sense

Summary: Since background knowledge and experience is a significant factor in the range and differentiation of interpretative frames (Klein, 2006), alternative perspectives can offer an advantage in sensemaking. Overly hierarchical teams can lead to the deferral of sensemaking by junior members and the failure to declare critical information. Develop strategies for taking soundings from team members and also those outside the team who may offer a fresh perspective. Strategies have been described for accessing and managing alternative interpretations, such as the appointment of a Devil’s Advocate (Klein et al., 2007; Klein et al., 2010; Hammond et al., 2006), setting rules ritualised dissent (Klein, 2010), setting criteria for alarm raising, a system of rewards for encouraging and reporting of errors (Rochlin, 1996) and make the most of any failure that is reported (Weick et al., 1999), review meetings and informal huddles (Attfield and Blandford, 2011). Different strategies may suit different situations.

​

Enhancing Sensemaking

Distributed Sensemaking and the Future Operating Environment

OFFICIAL

7. Where appropriate, use strategies for frame enumeration and elimination

Summary: Sensemaking frequently involves abductive reasoning (reasoning to the best possible explanation) (Klein et al., 2007). However, abductive reasoning is ‘defeasible’ insofar as its conclusions are not guaranteed, even if they may appear compelling. A frequent limiting factor is the small number of possibilities or ‘frames’ considered at a given time. Klein et al. (2007) speculate an upper limit of three. Confirmation bias involves giving serious consideration to only one. Where appropriate, use strategies for frame enumeration and elimination. Feltovich et al., 1984 refers to such a set as a ‘logical competitor set’ and uses this as a measure of expertise. Analysis of Competing Hypotheses (ACH) is an elimination strategy proposed by Heuer (1999) intended to mitigate against confirmation bias. Signal’s intelligence analysts have been observed to use elimination when drawing inferences from signals (Attfield et al., 2015). Such techniques frequently require the use of specially design artifacts or technologies that lay out a logical competitor set against a set of features and allow the recording of elimination decisions.

​

Enhancing Sensemaking

Distributed Sensemaking and the Future Operating Environment

OFFICIAL

• Empirical exploration of the issues associated with Distributed Sensemaking in a military context
• Exploratory first study – not hypothesis testing
• The study addresses open questions but is bounded by scope and duration
• To understand real-world military sensemaking we need expert participants
• Expert participants need a minimum of a complicated cognitive sensemaking challenge
• Therefore a MAPEX (Map Exercise without troops) was deemed suitable

Exploratory Study – Outline Requirement

OFFICIAL

• Explore principles 3, 6 and 7
• The study will use:
• A military scenario
• Military participants
• A dispersed operations context (i.e. using a STRIKE C2 scenario) and
• Provide sensemaking dilemmas at a Battle Group level

​

Summary

OFFICIAL

• Research Question 1 (RQ1): How do distributed groups negotiate distributed sensemaking problems?
• Research Question 2 (RQ2): How does an elaborated reporting format design impact upon distributed sensemaking?
• Research Question 3 (RQ3): What are the measurement issues associated with DSM?

​

Research Questions

OFFICIAL

• Three Company Commanders physically separated but connected by electronic media
• Communications comparable to radio and text systems
• Battle Group Commander is not present in the scenario
• Observer with each Company Commander
• Excon feeding each Company Commander their own unique reports

Experiment Set-up

OFFICIAL

Scenario

OFFICIAL

• Both qualitative data and communication transcripts for purposes of data analysis
• The participant audio will be recorded for the purpose of creating written transcripts
• The following general measures will be used based on observations:
• Observations targeting activity around sensemaking
• Temporal measures (e.g. time to complete; time to respond; etc)
• Performance measures (e.g. completion rates; errors/less good options taken)
• Middlesex Individual Sensemaking Questionnaire (SMQ) (Alsufiani, Attfield, & Zhang, 2017) after each vignette
• Semi-structured interviews as part of post-hoc debriefs with individuals and the team
• DSM structured Interviews guided by the DSM Principles (with the team)

​

Measures

OFFICIAL

Communications Analysis

OFFICIAL

Communications transcribed

Analysis of Communications network… Map of words used…

Words used by different speakers… Definition of speech acts

Next Steps

OFFICIAL

Benefits

• Provide a fuller understanding of the challenges of distributed sensemaking in military settings
• Provide a basis for improving coordination of purpose and shared understanding of distributed units

Next Steps

• Further research is planned to test and add to the distributed sensemaking principles
• The findings will be used to:
• Generate and test measures of good distributed sensemaking (including teams with synthetic team members)
• Develop interventions to improve distributed sensemaking such as training and system design

UK MOD © Crown copyright 2021

Discover more

Definitions for distributed sensemaking

• Distributed sensemaking unit (DSM unit): An individual, a semi-autonomous or autonomous system, or a collaborating group (of DSM units) who perform a sensemaking function. DSM units may be distributed spatially, temporally and/or by function.
• Distributed sensemaking (DSM): A process through which one or more DSM units seek to explain, understand, or otherwise make sense of a situation under uncertain or ambiguous conditions for informing a course of action. Each DSM unit seeks data and applies its values and goals, expertise and domain knowledge to synthesise data, draw inferences and share situation-pictures with other DSM units for collective understanding and co-ordinated action.

​

Task 1: Defining Sensemaking

OFFICIAL

• Attfield, S., Fields, B., Wheat, A., Hutton, R., Nixon, J., Leggatt, A., and Blackford, H. (2015). ‘Distributed sensemaking: a case study of military analysis’, International Conference on Naturalistic Decision Making, McLean, VA.
• Attfield, S. and Blandford, A. (2011) 'Making Sense of Digital Footprints in Team-Based Legal Investigations: The Acquisition of Focus', Human-Computer Interaction: Sensemaking, vol. 26, no. 1-2, pp. 38-71.
• Elliott, E., Attfield, S., Minocha, S., Fields, B., Hutton, R., Baber, C. (2020) ‘Enhancing Sensemaking: Supporting Distributed Groups in the Future Operating Environment. 25th ICCRTS.
• Hammond, J., Keeney, R. and Raiffa, H. (2006) ‘The hidden traps in decision making’, Harvard Business Review, vol. 84, no. 1, pp. 118-126.
• Heuer, R. (1999) Psychology of Intelligence Analysis. Central Intelligence Agency, Center for the Study of Intelligence, Washington, D.C.
• Klein, G. (2006) 'The strengths and limitations of teams for detecting problems', Cognition, Technology & Work, vol. 8, no. 4, pp. 227-236.
• Klein, G., Moon, B., & Hoffman, R. R. (2006a). Making sense of sensemaking 1: Alternative perspectives. IEEE Intelligent Systems, 21(4), 70–73.
• Klein, G., Phillips, J.K., Rall, E.L. and Peluso, D.A., (2007). ‘A data-frame theory of sensemaking. In Expertise out of context’. In Proceedings of the sixth international conference on naturalistic decision making, pp. 113-155. New York, NY, USA: Lawrence Erlbaum.
• Klein, G., Wiggins, S. and Dominguez, C.O. (2010) 'Team sensemaking', Theoretical issues in ergonomics science, vol. 11, no. 4, pp. 304-320.
• Rochlin, G. I. (1996) ‘Reliable organizations: present research and future organisations’, Journal of Contingencies and Crisis Management, vol. 4, no. 2, pp. 55-59.
• Weick, Karl E., Sutcliffe, K. M. and Obstfeld, D. (1999). ‘Organizing for High Reliability: Processes of Collective Mindfulness’, Research in Organizational Behavior, vol. 21, pp. 81-123.

​

​

References

OFFICIAL