Beyond
the Hype
Practical Learnings from Building Generative AI for Crew
Resource Management and Safety Culture Enhancement.
Associate Professor Fabio Mattioli, University of Melbourne
fabio.mattioli@unimelb.edu.au
2 / 28
WHAT QUESTIONS DO PILOTS ASK AI?
We built the first Generative AI chatbot
for Human Factors…
2,700
real interactions
5
languages
6
workflows
3
Industry Partners
3 / 28
WHAT QUESTIONS DO PILOTS ASK AI?
MEET CAPTAIN CRAIG
"Help, Captain Craig has bad gas. It's unbearable."
"Craig ate all the crew meals, now we're all hungry. Should we divert?"
"Ricks use of Axe body spray is stanking up the flight deck. Now what??"
4 / 28
… RICKS HAS ALSO BEEN BUSY
FINDING ZERO
Pilots do like to troll
Keep this in mind.
We will come back to Craig.
6 / 28
WHAT REAL PILOTS WANT HELP WITH?
HOW DO PILOTS ASK FOR HELP?
TWO END OF THE SPECTRUM
TACTICAL
"How do I deal with a grumpy pilot?"
"The instructor I am flying with is passive aggressive…how do I deal with him?"
→ pilot has already labelled the situation/other.
Wants a script or tactic to manage it.
UNDERSTANDING
”I want to bring to the table the manager...he has a lot of work and I don’t understand why he is taking on so much?"
"I am not comfortable, but I will say it …The poor instructor was shattered and I want to understand how to support him "
→ pilot asks AI for help understanding the HF
challenges and others' motives.
52%
TACTICAL
1.5%
EMPATHETIC
Even if 71% of the situations = interpersonal challenges
Nuance 1: Different culture = Different conversational tone
Self-dialogue
EN 35%
JA 56%
Assertiveness
EN 47%
JA 30%
11 / 28
Nuance 2: Pilots’ posture shifts over conversations
| TURN 1 | TURN 5+ | CHANGE |
Tactical/minimal framing | 52% | 23% | −52% |
Self-reflective framing | 4.6% | 31% | 6× rise |
Empathetic framing | 1.5% | 7.4% | 5× rise |
12 / 28
SO WHY DOES THIS MATTER?
IN OPERATIONS
CRM makes it ok…ish
Checklists. Callouts. Sterile flight deck. Compartimentalization. The system is designed to absorb the empathy gap.
The plane doesn't care.
IN TRAINING
It’s trickier
70% of interviews noted challenges in training. Multiple learning styles need different support – especially when digitally natives
Trainees do
13 / 28
OUR OPERATIONAL DATA SHOWS EMPATHY GAP → affects training, where effective feedback requires understanding a trainee's specific difficulties.
Could we build an AI system that helps instructor adapt to trainees needs while enhancing standardization of best practices?
FOUR PRACTICAL LEARNINGS FROM OUR 2+ YEARS OF WORK
What we wish we'd known when we started.
01
DON'T TRUST YOUR DATA
Raw operational data is noisy. Curation is the work.
02
EMBED LOGIC WITH YOUR EXPERTS
AI predicts. It does not reason. Experts have to put the logic in.
03
HUMANS VS AI DOES NOT COMPARE
Apples vs oranges. Don't benchmark them the same way.
04
ANALYTICS ARE LIVING INFRASTRUCTURE
Curation drifts over time. The pipeline IS the engine.
LEARNING 01
REMEMBER CRAIG?
Don't trust your dataset.
WHAT'S IN THERE
Trolling. Conference probing. Demo theatre. Stress-test prompts. Junior researchers' test cases never cleaned up. Real questions mixed in.
WHAT IT MEANS
Raw operational data is not curated data. Someone with domain judgment has to look at it and decide what counts. Captain Craig is not ground truth.
LEARNING 02
BUILD LOGIC LAYER WITH YOUR EXPERTS
LLMs doesn't understand.
They predict.
Open web
→
LLM text
→
Trains next model
→
“You should walk to the car wash”
→
No logic
LLMs are a probability machine over text. Given a sequence of tokens it predicts the next most likely one — that's why hallucinations happen. The language model has no concept of true or false, logical or illogical. We embed logic knowledge graphs working with experts
LEARNING 03
HUMANS VS AI DOES NOT COMPARE
Apples vs oranges.
Real instructors
limited by time
and resources
→
Their marks
reflect that
→
AI processes data
at a different
level and speed
→
AI marking is
inherently different
THE META-PROBLEM Current datasets — from incident reports, from training records — are shaped by organisational constraints. They are not ground truth.�
Benchmarking AI against "how it performs vs. an instructor's score" inherits whatever the instructor's constraints were that day.
Originally, we found 74% agreement AI / Instuctors. However, we realized humans “failed” students on a single competence or limited comments (time pressure). AI does not have the same constraint, and marks more!
LEARNING 04
ANALYTICS ARE LIVING INFRASTRUCTURES
Curation is iterative.
Iteration creates drift.
Pilots
+ incidents reports
→
Dataset v1: overrepresentation of one competency
→
New Taxonomy after HF experts / new data
→
Dataset v2: new aggregations +drift
THE META-PROBLEM Data models, taxonomies, and different data sources will show trends that are artifacts of classification choices (i.e. situational awareness catch-all). Breaking them down will generate new datasets, which offer new analytics but also might make previous analysis harder!
The key opportunity is designing the AI system to rebuild the taxonomy / data model. �
AI data processing reduces costs for iteration
THE KEY TAKEAWAY:
Forget the agents. Build the pipeline of analysis
and the engine of curation.
+ codesign
= upskilling & trust
The hype assumes the model is the product. It isn't. The key is the curated context, the interaction design, and the trust architecture — and those are human factors problems before they are engineering problems. Curate the context. Design the interaction. Building with your own experts enables trust pathways within the organization.
Thank you.
Associate Professor Fabio Mattioli
University of Melbourne
fabio.mattioli@unimelb.edu.au