1 of 12

Email: mohamed.abdelazez@forces.gc.ca

Extended Reality for Multi-Domain Operations

Mohamed Abdelazezand Ahmed Ghanmi

Defence Research Development Canada

Centre of Operational Research and Analysis

2 of 12

Multi-Domain Operation (MDO)

MDO involve using technologies to achieve functions like dissemination, discovery, collaboration, and resource management
Coordinating these operations is challenging, particularly in terms of achieving inter-domain effects
Crux of C2 lies in the ability of the commander to make informed decisions in an appropriate time frame to achieve the mission's objective

Image copied from: https://www.act.nato.int/article/mdo-in-nato-explained/

3 of 12

MDO Visualization

Elevated stress on the commander to make sense of varied and voluminous data
Presenting accessible data to the commander is crucial for sound decisions
TTCP TP44 study assignment to investigate the use of eXtended Reality (XR) technologies such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) to improve decision-making in MDO

Image copied from: https://www.lockheedmartin.com/en-us/products/command-and-control.html

Making informed decisions in a multi-domain environment is complicated by the amount of data that a commander must ingest and make sense of. Compared to a single domain, there will be a plethora of varied data from the multi-domains. It is crucial to present this data to the operators, analysts, and most importantly the commander to reach sound and informed conclusions. Therefore, under The Technical Cooperation Program (TTCP) Technical Panel (TP44) a study assignment was commissioned. The objective of the assignment is to investigate the use of eXtended Reality (XR) technologies such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) to improve decision-making in multi-domain operations. This presentation will report the findings of the study assignment and discuss the fundamentals of XR, current XR technologies, human factors, and military applications of XR from the literature and participating nations. It will also provide recommendations for future work.

4 of 12

XR Technologies

Extended Reality (XR) is an umbrella term for a range of immersive technologies, encompassing VR, AR, and MR
XR technologies can be classified on Milgram & Kishino’s Reality–Virtuality (RV) continuum seen below
Second vertical axis can be added to reality–virtuality continuum to illustrate that the interaction between the virtual and real worlds

Extended Reality (XR) is an umbrella term for a range of immersive technologies, encompassing VR, AR, and MR. These innovative solutions offer unprecedented opportunities for revolutionizing various industries, including military applications such as MDC2. XR technologies allow users to experience immersive environments that blur the lines between virtual and real worlds, enabling more effective decision-making in complex command and control scenarios. This section of the report defines these technologies and their relationships. XR technologies can be classified on Milgram & Kishino’s Reality–Virtuality (RV) continuum, as seen in the figure below. This continuum describes various technologies as some combination of the real environment and the virtual environment. A second vertical axis can be added to reality–virtuality continuum in Figure 1 to illustrate that the interaction between the virtual and real worlds is not present in the cases of augmented reality and augmented virtuality. At the same time, it is present in mixed reality.

5 of 12

Virtual Reality

VR encompasses immersive technologies that transport users into fully computer-generated environments
VR is characterized by stereoscopic displays, generating separate views for each eye, which is fused by the brain to enable depth perception
Benefits – Ego-centric frame of reference, safer, cheaper, and greener training, and improved visualization and comprehension
Risks – Cybersickness, musculoskeletal discomfort and injuries, physical harm, and psychological issues

Image copied from: https://www.simplilearn.com/tutorials/artificial-intelligence-tutorial/what-is-virtual-reality

VR encompasses immersive technologies that transport users into fully computer-generated environments. VR provides a fully immersive experience, isolating users from the physical world through head-mounted displays (HMDs). The military sector is among the potential beneficiaries of VR's capabilities. Commanders can utilize VR to visualize and analyze complex battlefield conditions and troop movements through detailed simulations. Additionally, remote collaboration between geographically dispersed teams becomes possible as members share a virtual space for efficient interaction.

The benefits of VR include:

1. Ego-centric frame of reference: Enhances performance for tasks requiring real-time control and manipulation of virtual environments, such as drone racing and remotely operating machinery.

2. Safer, cheaper, and greener training: This replaces real-world training scenarios in potentially dangerous or expensive situations, such as flight simulation, military training, etc.

3. Improved visualization and comprehension: This feature assists users in gaining insights about complex environments by allowing virtual walkthroughs, rapid prototyping, and immersive data analysis [22], [23].

The risks of VR are:

1. Cybersickness (CS): This is induced by linear and angular accelerations within a VR environment, resulting in mild symptoms like dizziness, discomfort, or severe nausea and vomiting. Symptoms usually disappear once exposure to the VR environment is stopped. Prolonged use or excessive motion can exacerbate CS [24].

2. Musculoskeletal discomfort and injuries: Prolonged use of VR headsets puts additional load on the neck, potentially leading to discomfort or injuries at the neck and shoulder levels [25].

3. Physical harm: Users may become so engrossed in the VR experience that they ignore their real-world surroundings, increasing the risk of accidents or injuries.

4. Psychological issues: Prolonged use of VR can lead to addiction, social isolation, and changes in perception of reality. These risks can be mitigated through proper design considerations for user interfaces, training programs, and guidelines on safe usage.

6 of 12

Augmented Reality

AR is another fundamental XR technology that overlays digital information into real-world environments, augmenting rather than replacing reality entirely
AR shares many of the benefits with VR, including an ego-centric frame of reference, safer training, and improved visualization and comprehension
AR also shares some of the risks with VR, though being able to see the real-world reduces the severity of some of these risks such as cybersickness and psychological issues
Additional benefits – better sense-of-presence within the real world and ability to situate visualizations

Image copied from: https://www.forbes.com/councils/theyec/2019/02/06/augmented-reality-in-business-how-ar-may-change-the-way-we-work/

AR is another fundamental XR technology that overlays digital information into real-world environments, augmenting rather than replacing reality entirely. AR systems could provide commanders with critical situational awareness by overlaying essential data in real time on their field of view. This could enable informed decision-making without constantly switching between various displays. In military applications, AR collaboration tools display relevant information and communications directly into a user's line of sight, fostering seamless teamwork and enhancing operational efficiency. AR is primarily enabled through two types of technologies: video-passthrough, where external cameras capture an image feed which is then presented to the user on opaque displays, and optical see-through, where a transparent display is used to augment the real-world from the user’s point of view. Video-passthrough display technologies can include handheld displays (e.g. smartphones and tablets), and HMDs such as the Meta Quest. Optical see-through can include fixed displays (e.g. tabletops and windows), see-through transparent HMDs such as the Microsoft Hololens 2, and Head-up Displays (HUDs). AR shares many of the benefits with VR, including an ego-centric frame of reference, safer training, and improved visualization and comprehension. AR adds additional benefits, including: 1) a better sense-of-presence within the real world, which may increase the comfort level and facilitate interactions between collocated users, and 2) the ability to situated visualizations, where data visualization may be closely embedding within a physical-referent in the real-world.

7 of 12

Mixed Reality

MR situates the virtual information within the real-environment through tracking and sensing technology, blurring the lines between virtual and real content
MR systems enable users to engage with digital content and real-world environments concurrently, offering a unique capability with significant potential for military applications
Since MR blends all the frontiers of the virtual and real world, it inherits the benefits and risks of VR and AR, with the added risk of greater social isolation due to the possibility of greater engagement

Image copied from: https://www.totalreality.nl/en/what-is-mixed-reality/

MR technology augments the real-world with integrated virtual information. Whereas AR overlays information, much like a HUD, virtual information in MR situates the virtual information within the real-environment through tracking and sensing technology, blurring the lines between virtual and real content. MR represents the cutting edge of immersive technologies, lying at the intersection of VR and AR. In command center scenarios, MR could be utilized for mission planning, training, and execution through immersive virtual environments. Furthermore, it could provide valuable insights into enemy capabilities, friendly force locations, and intelligence data, allowing informed decisions in complex and rapidly evolving situations. MR systems extend beyond AR by enabling interaction and manipulation of both the physical and virtual environment. Since MR blends all the frontiers of the virtual and real world, it inherits the benefits and risks of VR and AR, with the added risk of greater social isolation due to the possibility of greater engagement.

8 of 12

Technology

Head Mounted Displays
CAVE Audio Visual Experience Automatic Virtual Environment
Tables and Fish Tanks
Autostereoscopic Displays

Image copied from: https://www.totalreality.nl/en/what-is-mixed-reality/

There are multiple technologies to achieve XR and what is listed here are just a sample.

Head Mounted Displays (HMD) - HMDs were the first technology introduced for virtual reality, and they present distinct views to each eye, resulting in a combined spatial perception from stereoscopy and kinetic depth induced by viewpoint changes.

CAVE - CAVEs surround the user with stereoscopic images projected onto large screens, typically around 2.5m in size. In contrast to HMDs, CAVE displays provide the user with extensive stereoscopic views beyond their field of view.

Tables and Fish Tanks - These systems offer advantages over CAVE-type systems while providing benefits over HMDs. They can be categorized as immersive, tabletop, or desktop VR. Fish Tank VR is a solution where a stereoscopic display of monitor size is used.

Autostereoscopic Displays - Autostereoscopic displays attempt to provide viewers binocular vision on display surfaces without relying on head tracking or intermediary glasses. The benefits of lenticular display should be evident; there is no need to wear a headset and multiple people can view the same screen

A comparison of the features and capabilities of various XR technologies is provided in the Table shown on the right. The comparison is based on the following:

User Number and Location: Systems can accommodate single or multiple users, either remotely (R) or collocated (C). For example, CAVE-type systems work for multiple users but with distorted views for all but the tracked user.
Modality: Technologies are suited for VR, AR, and MR depending on their design.
Perception: Most XR systems use stereoscopic perception, but AR and MR can be achieved using 2D displays like tablets.
Long term user comfort: This feature covers physical, visual, and CS comfort for extended usage. Categories are low, medium, high based on the expected impact on the selection of the type of display.
Cost: Estimated cost for acquisition, deployment, and maintenance is represented by symbols as follows: $ (<10K$), $$ (10K-100K$), and $$$ (100K-$ low millions).
Transportability: Systems have varying ease of moving and setting up based on their size and complexity.

Maturity/availability: Our study covers commercially available and demonstrated prototype systems, with maturity indicating potential performance and availability indicating commercial status.

9 of 12

Human Factors

Traditional Desktop Approaches (e.g., Mouse)
Interactive Surfaces
Using Tracked Physical Objects
Haptic Devices
Mid-Air Gestures
Gesture Recognition
Gaze-Based Interaction
Virtual Reality Locomotion Techniques (e.g., Teleport, Room-Scale)
Brain-Computer Interfaces

Image copied from: https://www.cnet.com/videos/i-wore-the-future-with-openbcis-brain-sensing-vr-headset-galea/

XR technologies aim to blend reality and virtual world using wearable devices, image projecting devices, and stationary display devices. Interacting with these devices can elevate or complicate the user experience in XR by either providing a seamless intuitive interface or a clucky interface. Traditional desktop approaches, such as using a mouse, often struggle with the translation from 2D inputs to 3D actions, although interactive surfaces and mid-air gestures provide more naturalistic interactions but may lack precision. In more immersive settings, VR interaction techniques include haptic devices, gesture recognition, brain-computer interfaces, and gaze-based interactions, each evaluated on parameters like effectiveness and user engagement. Additionally, physical objects can be tracked and used to control virtual objects, offering a potentially more immersive alternative to traditional controllers by mimicking real-world interactions [46]. Various methods of interaction and locomotion in immersive environments, such as Point of Interest, Gamepad, Teleport, and Room-Scale, have also been studied. Teleport is a method that allows users to instantly move to a new location in the virtual environment by selecting a point within the scene. It is designed to reduce motion sickness by eliminating the need for continuous motion, which can disorient some users. While Room-Scale method allows users to translate real-world movement to the virtual world. While choosing the display and interaction technology, it is important to consider cybersickness, a condition like motion sickness that can occur when interacting with virtual environments. It includes symptoms such as dizziness, nausea, and disorientation. These symptoms can arise from mismatches between visual motion cues and the lack of corresponding vestibular responses in a static real-world environment.

10 of 12

Military Applications

AFRL:

Project Quartz from the Air Force Research Lab aims to develop a collaborative planning tool based on MR. Multiple headsets were tested and a mixed reality view of terrain, targets, and threats was developed

DRDC:

As part of the TTCP Human System Performance (HUM) Joint Panel–Maritime (JP2), DRDC has been exploring the potential benefits of 3D visualization and integration of data for Antisubmarine Warfare (ASW)
Fundamental research on benefits of immersive technologies on decision making and cybersickness mitigation

DSTL:

Fundamental research is being conducted by Dstl on the benefit of immersive technologies on C2 at the operational level of command
The UK is also developing early concept work to explore and create bespoke overlays to augment static maps with information at the tactical level, for control and planning, within the Maritime Domain

DSTG:

Investigated using Mixed Reality (MR) in Agile Command and Control (C2) under a DSP research agreement

Image copied from: https://www.totalreality.nl/en/what-is-mixed-reality/

XR is currently being actively researched with new consumer products being released such Apple Vission Pro and the Ocuclus Rift providing advances in control of virtual environments and comfort. Militaries across the FVEY have explored the technology including for C2. In the US, project Quartz from the Air Force Research Lab aims to develop a collaborative planning tool based on MR. Multiple headsets were tested and a mixed reality view of terrain, targets, and threats was developed. The TTCP Human System Performance (HUM) Joint Panel–Maritime (JP2) has been exploring the potential benefits of 3D visualization and integration of data for Antisubmarine Warfare (ASW) command teams [50]. Recent advancements in XR technologies have made it possible to present relevant ASW data in 3D. A series of demonstrations were held, where XR ASW prototypes were presented to ASW subject matter experts (SMEs) from the Royal Canadian Navy (RCN), the United Kingdom’s Royal Navy (RN), and Defence Research and Development Canada (DRDC). In the UK, fundamental research is being conducted by Dstl on the benefit of immersive technologies on C2 at the operational level of command . The objective of this research is to understand “whether 3D immersive technologies enhance individual situation awareness and sensemaking, collaborative situation awareness and sensemaking, and individual collaborative decision making”. The UK is also developing early concept work to explore and create bespoke overlays to augment static maps with information at the tactical level, for control and planning, within the Maritime Domain. While, DSTG investigated using Mixed Reality (MR) in Agile Command and Control (C2) under a DSP research agreement. A literature review was conducted, categorized within the OODA framework's Observe, Orient, Decide, and Act processes. The review highlights the lack of multi-domain studies on MR C2 environments' effects.

11 of 12

Conclusion

XR remains an actively research field specifically as a technology for the military. Here are some future projects that would benefit the member nations from exploring:

Large scale experiment using military members to establish the impact of XR technologies on decision making, situational awareness, collaboration, and information sharing. Such an experiment has not been conducted and the magnitude of the impact of XR on C2 and MDO is yet to be defined.
Investigation of data visualization based on data and display types including 2D and 3D technologies.
Investigation of trust of human-human and human-machine interactions within XR for collocated and distributed C2.
Investigation of the use of generative AI as a virtual assistant to support different C2 and MDO functions.

MDC2 in modern warfare highlights the need for a comprehensive system that provides situational awareness, decision support, communication interoperability, security, scalability, user-friendly interface, human-machine teaming, mobility, reliability, data fusion and visualization, training, and simulation. XR technologies, including VR, AR, and MR, may enhance decision-making in complex command and control situations. XR remains an actively research field specifically as a technology for the military. Here are some future projects that would benefit the member nations from exploring:

- Large scale experiment using military members to establish the impact of XR technologies on decision making, situational awareness, collaboration, and information sharing. Such an experiement has not been conducted and the magnitude of the impact of XR on C2 and MDO is yet to be defined.

- Investigation of data visualization based on data and display types including 2D and 3D technologies.

- Investigation of trust of human-human and human-machine interactions within XR for collocated and distributed C2.

- Investigation of the use of generative AI as a virtual assistant to support different C2 and MDO functions.

12 of 12

Email: mohamed.abdelazez@forces.gc.ca

Extended Reality for Multi-Domain Operations

Mohamed Abdelazezand Ahmed Ghanmi

Defence Research Development Canada

Centre of Operational Research and Analysis