1 of 23

27th International Command and Control Research and Technology Symposium, Quebec City, Oct 2022

A Data Collection Approach for Evaluating National Security Collective Training Events

Jerzy Jarmasz

Defence R&D Canada, Toronto Research Centre

​

Michael Thomson, Jonathan Fugelsang, Evan Risko

HumanSystems Inc.

​

​

CAN UNCLASSIFIED

CAN UNCLASSIFIED

2 of 23

Overview

  • Overview of Exercise Global Archer 2020
  • Measurement challenges
  • Approach
  • Review of data collection
  • What did we learn?
  • Next steps

​

2

CAN UNCLASSIFIED

CAN CLASSIFICATION // REL TO

3 of 23

Overview of the Exercise: Global Archer 2020 (GA20)

  • Collective “field exercise” Training Event, led by Canadian Special Operations Forces Command (CANSOFCOM) in 2020.
    • Expedited National Security (NS) crisis incident scenario
    • NS partners collaborated to develop coordinated, Whole of Government (WoG) response to a potential National threat in a very compressed timeframe
    • Activities at Tactical, Operational and Strategic levels
    • Decisions centred on determining authorities (legal and functional) and roles to enable expedited, lawful, pan-domain response to the threat

3

CAN UNCLASSIFIED

CAN CLASSIFICATION // REL TO

4 of 23

Measurement Challenges

  • No a priori “right way” of achieving strategic objectives
    • No objective criteria for inter-agency activity effectiveness
  • No direct access to objective activity data (e.g., comms logs)
  • “Field exercise” nature of event:
    • Varying degrees of access to activities across Strategic, Operational and Tactical levels
    • Varying degrees of access to partner organizations other than DND/CAF
  • Real-life requirements led to real-time scenario changes

4

CAN UNCLASSIFIED

CAN CLASSIFICATION // REL TO

5 of 23

The approach

  • Focus on Inter-agency activity at Op/Strat Levels
  • Triangulation: combine types of measures for multiple perspectives
  • Draw on various literatures:
    • C2 (esp. in WoG/pan-domain contexts)
    • Naturalistic decision making
    • Multi-team systems
    • Collective training design and assessment

​

5

CAN UNCLASSIFIED

CAN CLASSIFICATION // REL TO

6 of 23

The measures developed

  • Survey instrument
    • Post-exercise questionnaire, administered to all
  • Adapted Social Network Analysis (SNA) method
    • Probing perceptions of information sharing, pre & post exercise
  • Semi-structured interviews (supporting SNA)
  • Informal observations at planning/lead up events and GA20 itself

6

CAN UNCLASSIFIED

CAN CLASSIFICATION // REL TO

7 of 23

Post-Exercise Questionnaire

  • 4 Survey themes, developed with CANSOFCOM:
    1. WoG coordination and Planning
    2. Exercise conduct
    3. Exercise fidelity and realism
    4. Exercise value and effectiveness
  • Was sent to all participating GC entities by CANSOF; responses were encouraged but voluntary (27 returns received from 9 orgs)
  • Only descriptive stats analyzed; number of responses did not support statistical comparisons

7

CAN UNCLASSIFIED

CAN CLASSIFICATION // REL TO

8 of 23

Sample questions

8

CAN UNCLASSIFIED

CAN CLASSIFICATION // REL TO

9 of 23

SNA instrument

  • SNA methodology probing information sharing
    • Adapted by asking about perceptions of info sharing (objective data not accessible)
    • Asked variety of questions: was info shared, was it timely, relevant etc.
  • Asked questions pre & post (e.g., what info sharing did you expect vs. what did you experience)
  • Responses collected with 3-pt rating scale
  • Various metrics (e.g., frequency, centrality) computed
  • Data collected from 6 consenting orgs
    • DND/CAF entities counted as one org in the following analyses
  • Wide range of analyses possible; only select few presented here due to time constraints

9

CAN UNCLASSIFIED

CAN CLASSIFICATION // REL TO

10 of 23

SNA questionnaire sample

10

In response to a national security crisis, my organization would typically send information to [or: rely upon, or expect info from]:

 

Org 1

Org 2

…

Org X

Not at all

Coded as 1

 

​

 

Infrequently

Coded as 2

 

​

 

Frequently

Coded as 3 

 

​

 

During the Ex, my organization sent information to:

 

Org 1

Org 2

…

Org X

Not at all

 

 

​

 

Infrequently

 

 

​

 

Frequently

 

 

​

 

Sample Pre-exercise question:

Sample post-exercise question:

CAN UNCLASSIFIED

CAN CLASSIFICATION // REL TO

11 of 23

Big picture: Caveats and Findings

  • Some important caveats:
    • Small # of respondents (limits statistical validity)
    • Measures not (yet) validated
    • Mostly subjective measures
    • Key decision making events in the scenario had to be scripted out due to real-world operational requirements
  • Data shed light on inter-agency activity as a whole, but evaluation of any single org, or comparison of performance of participating orgs in GA20 to be avoided.

​

11

CAN UNCLASSIFIED

CAN CLASSIFICATION // REL TO

12 of 23

Survey findings: Theme 1 WoG Coordination and Planning

12

To what extent did…

Mean rating

SD

Q1: inter-departmental info sharing effectively support WoG coordination and planning?

3.47

1.3

Q2: current governance frameworks (e.g., FTRP) and policies support WoG cordination and planning?

3.05

1.24

Q3: the departments effectively balance the need to protect vs. the need to share information?

3.78

1.24

Q4: departments develop a shared understanding of the governance structure?

3.20

1.12

Q5: departments develop shared Situation Awareness during the exercise

3.34

1.47

Q6: departments develop coordinated objectives?

3.35

1.44

Q7: the national security partners effectively coordinate the execution of their activities?

3.79

1.3

Theme Overall

3.46

0.89

Internal consistency (Cronbach’s Alpha): 0.94

CAN UNCLASSIFIED

CAN CLASSIFICATION // REL TO

13 of 23

Survey findings: Theme 2 Exercise Conduct

13

To what extent were…

(* = reverse coded post-hoc)

Mean Rating

SD

Q1: exercise injects disseminated appropriately (e.g., on time, to the right players)?

3.84

0.99

*Q2: blue Force objectives hindered by ineffective inject dissemination or reception?

4.07

0.59

Q3: requests for Information (RFIs) made by Blue Force players to other Blue Force players justified?

4.09

1.66

Q4: requests for Information (RFIs) made to Blue Force players responded to appropriately?

3.50

1.78

Q5: requests for Information (RFIs) made by Blue Force players to EXCON justified?

3.75

1.95

Q6: backstops employed by Trusted Agents to ensure the good progress of the scenario?

3.87

0.96

Theme Overall

3.86

0.66

Internal consistency (Cronbach’s Alpha): 0.82

​

​

CAN UNCLASSIFIED

CAN CLASSIFICATION // REL TO

14 of 23

Survey findings: Theme 3 Exercise fidelity and realism

To what extent did…

(* = reverse coded post-hoc)

Mean rating

SD

Q1: the governance structure employed in the scenario represent a plausible governance structure for an actual NS crisis incident

4.05

1.25

Q2: the information sharing protocols employed in the exercise reflect real GC protocols (e.g., FTRP) and their use?

3.72

1.53

Q3: the scenario support the use of inter-departmental or WoG information sharing, situation awareness or planning systems/tools?

4.05

1.43

*Q4: the exercise impose artificial challenges or frictions on the Blue Force players?

3.70

0.82

Q5: the exercise capture important real-life challenges that would face the WoG apparatus during a NS crisis incident?

4.38

0.65

Q6: the exercise involve the staff and planners who would or should be involved in a real NS crisis incident?

3.92

1.27

Q7: the exercise involve the decision makers (e.g., ADMs or above) who would or should be involved in a real NS crisis incident?

3.31

1.84

Q8: the scenario reflect a realistic level of complexity for NS crisis information sharing and decision making?

4.38

0.71

Q9: the scenario capture the stress of real NS WoG activities?

3.67

1.15

*Q10: the various planning conferences provide an artificial “heads up” to the Blue Force?

3.06

1.39

Theme Overall

3.81

0.86

Internal consistency (Cronbach’s Alpha): 0.23

CAN UNCLASSIFIED

CAN CLASSIFICATION // REL TO

15 of 23

Survey findings: Theme 4 Exercise value and effectiveness

15

To what extent did the exercise, and/or its planning over the last 18 months…

Mean rating

SD

Q1: improve your department’s understanding of WoG governance during a NS crisis incident?

3.55

1.20

Q2: improve your department’s ability to support WoG decision making during a NS crisis?

3.68

1.43

Q3: enable departments to identify key areas for improvement in their ability to support WoG NS crisis planning?

4.00

1.43

Q4: improve WoG information sharing?

3.38

1.63

Q5: improve WoG common situation awareness?

3.63

1.64

Q6: improve WoG planning and objective setting?

3.57

1.60

Q7: improve WoG execution of coordinated effects?

3.53

1.49

Theme Overall

3.81

0.86

Internal consistency (Cronbach’s Alpha): 0.93

CAN UNCLASSIFIED

CAN CLASSIFICATION // REL TO

16 of 23

SNA send/receive ratings

16

Sub-organization

Pre

Post

Correlation

DND

2.3

2.3

0.64

RCMP

2.3

2.1

0.71

ITAC

2.8

1.4

0.2

IRCC

2

1.3

0.24

FINTRAC

1.7

1.4

0.60

Sub-organization

Pre

Post

Correlation

DND

2.5

1.8

0.45

RCMP

2.3

2.1

0.70

ITAC

2.8

2.1

0.51

IRCC

1.8

1.3

-0.16

FINTRAC

1.8

1.8

0.68

Mean info “would send to” (Pre) and “we did send to” (post) ratings and correlations

Mean info “would receive from” (pre) and “we did receive” (post) ratings and correlations

CAN UNCLASSIFIED

CAN CLASSIFICATION // REL TO

17 of 23

Alternate metric: SNA centrality graphs (1/2)

17

Comparison between sub-organization’s expectations about who they need to send information to both pre-exercise (left panel) and post-exercise (right panel)

Note: node size and colour reflects centrality, i.e., larger and darker nodes reflects higher centrality; line weights and colour reflects the expected frequency with which information is expected to be sent to a given sub-organization, i.e., heavier and darker lines reflect more frequent sending of information.)

CAN UNCLASSIFIED

CAN CLASSIFICATION // REL TO

18 of 23

Alternate metric: SNA centrality graphs (2/2)

18

Comparison between sub-organization’s expectations about who they need to receive information from both pre-exercise (left panel) and post-exercise (right panel)

Note: node size and colour reflects centrality, i.e., larger and darker nodes reflects higher centrality; line weights and colour reflects the expected frequency with which information is expected to be sent to a given sub-organization, i.e., heavier and darker lines reflect more frequent sending of information.)

CAN UNCLASSIFIED

CAN CLASSIFICATION // REL TO

19 of 23

Example of possible additional analysis: �Alignment of info sharing

19

Alignment of reported info sharing expectations (pre-exercise) between select participating sub-organizations

Each arrow represents alignment of perceptions by two organizations of the information flow from the first to the second:

Red solid line: misaligned

Blue dashed line: moderately aligned

Green dashed line: well aligned

​

While the earlier caveats on drawing inferences comparing organizations apply, this analysis illustrates a metric that could be used to monitor expectations between participants during an exercise.

CAN UNCLASSIFIED

CAN CLASSIFICATION // REL TO

20 of 23

Discussion: Insights about GA20

  • Observed high willingness for inter-agency collaboration, as well as a large amount of inter-agency activity
    • reflected in data and informal observations
    • Inter-agency activity occurs at all levels, not just Strat or Op/Strat; many interlocking loops of activity
  • The planning & work-up activities were instrumental in priming this collaboration
  • Flexible scenario design allowed the Ex to proceed despite real-life interference

20

CAN UNCLASSIFIED

CAN CLASSIFICATION // REL TO

21 of 23

Discussion: Insights about Data Collection

  • Post-Ex survey design: all themes except one had robust internal consistency
    • Theme 3 needs a review; but the questions overall can serve as basis for future exercise evaluations
  • SNA is a very promising methodology; its full potential remains to be explored
    • While objective data on info sharing would have been preferable, adapted SNA allows investigation of dynamics of expectations and “mental models” of inter-agency activity
  • Using multiple metrics worked well
    • Ensured we got some useable data despite real-time scenario changes
    • Provided multiple perspectives on complex event

​

21

CAN UNCLASSIFIED

CAN CLASSIFICATION // REL TO

22 of 23

Next steps

  • What we’d like to do better
    • Revise & improve consistency of existing measures
    • Improve response rates for data collection
    • Improve alignment between scenario and data collection
  • For future exercises
    • Build on existing measures, validate what we can
    • Examine what other dimensions (e.g., support to strategic decision making) could be measured
    • Explore options for objective measures

22

CAN UNCLASSIFIED

CAN CLASSIFICATION // REL TO

23 of 23

Questions?

​

Contact: jerzy.jarmasz@drdc-rddc.gc.ca

23

Thank you

CAN UNCLASSIFIED

CAN CLASSIFICATION // REL TO