3 of 25

Purpose of Study

To explore the experience of MQ-4C Triton Air Vehicle Operators (AVO) to build theory about the UAS characteristics that affect the handling of the system.

Research Questions

What are the perceptions of AVO of their role in UAS operations?
What difficulties do AVO experience when operating UAS?
How do AVO perceive the effect of UAS design on operating a UAS?
What, if any, strategies or techniques do AVO use to overcome system limitations?

“Exploration of the Handling Qualities of an Autonomous

Unmanned Aircraft System Using a Grounded Theory

Methodology Toward the Identification of

Characteristic Traits” –dissertation (McLean, 2022)

4 of 25

Methodology

Grounded Theory methodology

Introduced by Glaser & Strauss in 1967

Subset of Phenomenology

Utilizes inductive reasoning to build theory

No established Hypothesis to prove or disprove

Semi-structured Interviews

Discussions that allow the participant to expand on comments and insights
Guide provided to foster the discussion, but not intended to constrain topics
Participants may be asked about the comments or insights of others

Everything is data

Other forms of data may be included for analysis (Mishaps, training material, etc.)

Robust analytic process through Constant Comparison

Everything is compared to everything else repeatedly
Identify patterns, build categories from the patterns
Categories are used to build theory
Analysis continues until saturation is reached

5 of 25

Participants

Eleven qualified MQ-4C Triton Air Vehicle Operators (AVO)

Two generic groups:

Flight test group, Triton Integrated Test Team (ITT);
Operational group, Unmanned Patrol Squadron 19 (VUP-19)

All participants had previous manned-flight experience, with at least 1,000 flight hours in multi-crewed Naval manned aircraft.

Range of experience in MQ-4C

Newly qualified to multiple years

Participants recruited through their commands

Commands were not informed of individual participation

All willing volunteers were interviewed
Three participants were female
No race, ethnicity, or age demographics were collected
Adhered to UND IRB human subject requirements

IRB results reviewed and approved by Naval Air Warfare Center Aircraft Division

Interview	MQ-4C flight hours	Role	MPRA background	Other UAS experience
A	<50	Flight Test	No	Multiple
B	<50	Flight Test	Yes	No
C	>200	Flight Test	Yes	Multiple
D	>200	Flight Test	No	Multiple
E	>200	Flight Test	Yes	Multiple
F	>200	Flight Test	Yes	No
G	50-200	Operational	Yes	No
H	50-200	Operational	Yes	No
I	>200	Operational	Yes	No
J	>200	Operational	Yes	No
K	<50	Operational	Yes	No

6 of 25

MQ-4C Triton

High-altitude, long-endurance, air vehicle
Max takeoff weight: 32,250lb, Wingspan: 130ft
Beyond line-of-sight SATCOM datalinks
High-level of automation, inclusion of autonomous actions
Mouse and Keyboard interface
No direct control of the vehicle by the AVO

7 of 25

Flying Qualities��

System response to inputs (Kylde, 2020)

Attribute of the UAS

Human Factors��

Multidisciplinary effort to generate and compile information about human capabilities and limitations and apply that information … for safe, comfortable, and effective human performance(FAA, 2005)

“human factors issues in manned aviation are well known … there needs to be further analyses regarding UAS integration into the NAS” (FAA, 2013, p.30)

Handling Qualities��

“Those qualities or characteristics of an aircraft that govern the ease and precision with which a pilot is able to perform the tasks required in support of an aircraft role.” (Cooper & Harper, 1969)

9 of 25

Command

Triton Air Vehicle Operator is in command of the vehicle

Responsible for its safe operation (Aviate, Navigate, Communicate)
Not just a passive monitor
Requires information about the state and intent of the vehicle
Understanding of when to intervene

AVO is accountable for the mission effectiveness and safety of the system
Drives need for awareness and decision making.

10 of 25

AVO Tasks

Mission Task Elements are similar to manned aircraft

UAS in the NAS are expected to operate in accordance with existing procedures (use of existing waypoints, holding, flying assigned vectors).

Staying Ahead of the Robot

Tasks focused on understanding and interacting with the automation
Monitoring UAS performance
Preparation for autonomous contingencies
More mental workload than physical
Mouse and Keyboard prevent reverting to direct control

AVO is in command not in control

11 of 25

Flying Qualities

Traditional flying qualities do not apply

“UAVs have all of these classical flight modes, like they are going to Phugoid. Like every plane, it’s got a short period – pitch pointing mode. They all have it. But, how much does it matter to you as the operator. Most of the time it really doesn’t”. (AVO Interview)

Concept of flying qualities is still relevant.

Instead of being influenced by the qualities that define the control of the vehicle, they are defined by the system’s response to the commands sent by the AVO.

Flying qualities involve coordination between the AVO and UAS (exercise of command vice control).

Intuitiveness, Predictability, Flexibility

“Some of the easy stuff became hard, and in exchange some of the hard stuff became really really easy” (AVO interview).

12 of 25

Flying Qualities –Intuitive� - How closely the system’s response matches the AVO’s intent.

UA doesn’t always fly like a pilot would

“If your controls laws are closer to how a manned aircraft would react that would make more sense to you as a pilot” (AVO interview).

Fly-by vs Fly-over (Aeronautical Information Manual, 2011)

Aircraft descends in response to a commanded higher altitude

“If you want to climb and you are in endurance mode, yeah, it's going to descend to get to its nominal airspeed first and then climb. Well, in a manned airplane, we don't… we don't do it that way. We set power and start climbing” (AVO interview).

Drives AVO input shaping to achieve desired response

13 of 25

Flying Qualities – Predictability

The consistency of the response to the AVO’s intention

Goto waypoint with imbedded altitude command

Drives need to monitor vehicle response to commands

“The phrase we use is that this thing is the world’s worst flight lead. Because part of the effect of taking the pilot a little further out of the loop and just saying ‘hey here is what I want airplane, you do it’ is sometimes it will determine that instead of turning right to get somewhere, it is one degree shorter to turn left” (AVO interview)

“You say hey, go to this waypoint on the map, which is great. [However], you still, if you have a lock in, it is just going to grab all of the attributes of that waypoint and send it to the airplane. So, you have to go back to the Primary Flight Display and say ‘no no, I want you at this altitude not 50,000, I need you at 39,000’… and because you graphically said ‘go to this way point’ on the map, it bashes everything and blows off your lock. It goes there, and the next thing you know ATC is telling you your 10,000 ft above your altitude. Like crap. There are so many little gotchas” (AVO interview)

14 of 25

Flying Qualities – Flexibility

“The turn rate, because you can't change your angle of bank… the turn radius is completely unpredictable… Because, by their design, you can take it off the mission plan, where it has all the calculations that go into how to turn. But now that you've taken it off of the mission plan, which you may need for traffic avoidance, or for the mission or for whatever, you now have no way to… increase your turn radius or decrease your speed to decrease your turn radius. And it's completely out of your awareness of like what it's going to be and how you can maneuver it. So, I thought that was a real difficulty” (AVO interview).

“For turning I am limited to 15 degrees angle of bank, at altitude. We have had 150 knot crosswind before in R4008. Boy, and when you are dealing with some override steering commands, doing some ‘handling qualities’, I tell you what, you turn down wind… 14 miles later you are out of the airspace. So those things make it a challenge” (AVO interview).

“On a mission plan, those are self-explanatory, and you just have to keep up with every single possible contingency that the aircraft can go. When you're off mission plan, then that's when things start to possibly be more varsity as to what you are going to do” (AVO interview).

- The ability to command the UA to the desired state.

- Impacts mission effectiveness.

15 of 25

Human Factors

AVO are still human

Principles of predictive aiding, Discrimination, Information need, Display Integration/Proximity all seen in responses

Influence of attention, workload, compliancy, loss of environmental cues all noted
Data availability, dissemination, and quantity are all affected

Established Human Factors principles should be applied to the tasks performed by the AVO.

16 of 25

Human Factors - Interface

With different tasks and flying qualities, the function of the interface is different

Loss of environmental cues
Different tasks
Interaction with automation
Link status

Current manned standards do not adequately serve needs of AVO

Standard T not needed to complete AVO tasks
Still appreciated by AVOs (familiarity?)

UAS should facilitate the tasks assigned to the AVO

Spoiler deployment during engine out

17 of 25

Human Factors – Data Overload

Additional data to replace the loss of sensed information.

“If you look at like an Airbus, or like a 737, like they can display to you the same data, but like they don't… So, there's a reason that they don't want to display that much information to you because it's overwhelming. It’s not something that you take into account… UAS you have to be intentional about what you're going to show the operator. Now, it's great if you really need to diagnose something to have more information. But just because you can doesn't mean you should, just because it's too much information to process” (AVO interview)

Excess data can make it difficult to interpret what or if actions are required.

“You have a major subsystem failure and it trips 30 cautions and warnings, like what are you supposed to do with that? .... That is the biggest bane, the human factors engineering and the interfaces make it so bloody difficult, especially when things go wrong” (AVO interview).

“If you get compounding malfunction or cascading faults, that quickly becomes a can of worms” (AVO interview).

“How much information does a pilot need? The short answer is just enough” (Martinussen & Hunter 2017, p 192).

18 of 25

Human Factors - Engineering Speak

“[The difficulty] is coming at it from a pilot perspective when it was coded by engineers who don’t fly airplanes”

(AVO Interview).

“Instead of [showing] ground safety pin [in or out] … it's …some engineering terms equals false. Like there's a circuit in there, then you put [the safety pin] in, and [the message displayed to the AVO] is this closed relay circuit two equals false... You know, it's great for the guy that built the circuit, but not good for the AVO” (AVO Interview).

19 of 25

Handling Qualities - Automation as Copilot

The factors that influence the interaction are similar to the factors that affect Crew interactions.

AVO interaction with UAS should be framed as crewmembers

Several interviews made analogy to Plane commander and flying pilot or Instructor and student pilot.

Follows Dekker approach that frames design consideration for autonomous systems as “how do [the pilot and automation] get along together” (1994).
Potential application of existing Crew Resource Management principles to UAS design

20 of 25

Handling Qualities - Trust

With interaction between AVO and UAS framed in terms of teamwork, Trust emerges as a pertinent characteristic

AVO is shifted to tasks of monitoring UAS, Trust enables the AVO to reduce the workload spent on monitoring

Influenced by alerting system
Understanding of nominal response

More Trust means less effort operating the system

Potential capacity to operate multiple aircraft

21 of 25

So What?

Vinciti’s seven elements of the epistemological structure of the Flying Qualities learning process:

1. Familiarization with vehicle and recognition of problem.

2. Identification of basic variables and deviation of analytical concepts and criteria.

3. Development of instruments and piloting techniques for measurements in flight.

4. Growth and refinement of pilot opinion regarding desirable flying qualities.

5. Combination of partial results from 2, 3, 4 into deliberate, practical scheme for flying qualities research.

6. Measurement or relevant flight characteristics for a cross section of aircraft.

7. Assessment of results and data on flight characteristics in light of pilot opinion to arrive at general specification.

22 of 25

Autonomous UAS Handling Qualities

Different from manned / piloted aircraft

No longer a function of modes and control forces

Need to identify system characteristics that affect AVO evaluation

What makes an operator Trust the system?
How do you measure Trust or Teamwork?
Cooper Harper appears relevant

Standardization of system evaluation criteria and methods

How does the operator background affect the evaluation

Pilot vs AVO vs Untrained

23 of 25

UAS Taxonomy

Handling qualities influence the design standards required to ensure safe and effective aircraft.
The handling qualities of Triton are clearly different from other piloted aircraft and affected by the automation incorporated.
A taxonomy should be established that adequately differentiates aircraft based upon automation levels and control types.

SAE Levels of Driving Autonomy (SAE International, 2021)

UAS Categories (NATOPS General Flight and Operating Instructions Manual, 2016)

25 of 25

References

Baughman, C., & Longeauay, K. (2015). Handling Qualities Evaluations of Unmanned Aircraft Systems (No. 412TW-PA-15214). AIR FORCE TEST CENTER EDWARDS AFB CA TEST WING (412TH)/TEST PILOT SCHOOL.
Cooper, G. E., & Harper Jr, R. P. (1969). The use of pilot rating in the evaluation of aircraft handling qualities. NASA Technical Note. TN D-5153.
Cooke, N. J. (2006, October). Human factors of remotely operated vehicles. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 50, No. 1, pp. 166-169). Sage CA: Los Angeles, CA: SAGE Publications.
Crotty, Michael (1998) The foundations of social research. Sage Publications.
FAA. (2005). Human Factors Policy. Order 9550.8A.
FAA. (2013). Integration of civil Unmanned Aircraft Systems (UAS) in the National Airspace System (NAS) roadmap. Retrieved from https://www.faa.gov/uas/media/uas_roadmap_2013.pdf
FAA. (2017). General Aviation Fatal Accident Rate. ga_fatality.pdf (faa.gov)
FAA. (2020). Type Certification of Certain Unmanned Aircraft Systems. Notice of Policy. Docket No. FAA-2019-1038.
Hobbs, A., & Lyall, B. (2016). Human factors guidelines for remotely piloted aircraft system (RPAS) remote pilot stations (RPS). NASA Contractor report.
Hodgkinson, J. (1999). Aircraft handling qualities. Oxford.
NATOPS General Flight and Operating Instructions Manual, CNAF M-3710.7. (2016). https://www.cnatra.navy.mil/tw6/vt10/assets/docs/training/cnaf-3710.7.pdf.
Naval Safety Center. (2020). Mishap Statistics. Published 12 Aug 2020. Mishap_Stats081220.pdf (navy.mil)
SAE International. (2021, April). Taxonomy and definitions for terms related to driving automation and systems for on-road motor vehicles. SAE J3016.
Saldaña, J. (2021). The coding manual for qualitative researchers. SAGE Publications Limited.
Teal Group. (2020). Teal Group Predicts Worldwide Civil Drone Production Will More than Triple Over the Next Decade Despite Pandemic. 06 October 2020.
United States Government Accountability Office. (2008). Unmanned Aircraft Systems: Federal actions needed to ensure safety and expand their potential uses within the National. Report number GA)-8-511. May 2008.
Waraich, Q. R., Mazzuchi, T. A., Sarkani, S., & Rico, D. F. (2013). Minimizing Human Factors Mishaps in Unmanned Aircraft Systems. Ergonomics in Design, 21(1), 25–32.
Williams, K. W. (2004). A summary of unmanned aircraft accident/incident data: Human factors implications. Federal Aviation Administration Oklahoma City OK, Civil Aeromedical Inst.

1 of 25

2 of 25