1 of 42

Living in a Generative World

Barclay R. Brown, Ph.D.

Senior Technical Fellow, AI Research

Collins Applied Research and Technology (ART)

2 of 42

Welcome to the show!

This is going to be already big
It is for everyone, but not everyone knows it yet
Embrace it and grow with it

Image generated by the author

3 of 42

What if everything we want to create can be generated?

Song generated by the author

Play on Suno.com

4 of 42

It’s a world where…

Conductors are the players
Directors are the actors �(and sets)
Architects are the designers
Software architects are the coders
The score is a program

What is a Generative World?

image generated by the author

5 of 42

Descriptive Clarity

Clear, complete, unambiguous description

Includes prompt engineering, but more creative

Expect iterative refinement

It takes work to fully describe (longer prompts)

Direct to Create

Creation happens in the mind

Creativity but detailed, precise enough to implement

Assume only general knowledge

Invent and create in one seamless activity

Skills for a Generative World

Architecture

Specific Structure

Specific Behavior

Imagine Specifically

All images generated by the author

6 of 42

A Right-Now Experiment:��Hobson

Email Hobson a question or request about anything. He has general knowledge about everything, but no inside, company, or secret information about anyone or anything.

Send an email to:

Barclay.brown@collins.com

Then, in the body:

Hi Hobson (or Dear Hobson, Hey Hobson, Yo Hobson)

Your request, question, instructions, etc. You can attach .docx, .pdf, .txt, .csv, .py files

Don’t go too easy on Hobson!

Plan a vacation trip to Portugal
Write a poem about a famous battle
Summarize an attached document
Present the best arguments for or against a topic

Creating Hobson

LLM running on my personal workstation (Phi3 Medium LLM)

A paragraph or so about who he is (modeled after British Valet to Arthur in the original movie, played by John Gielgud, extremely polite, helpful, occasionally snarky, etc.)

6

7 of 42

Our Relationship to

Language-Based AIs?

In search of a metaphor

Horse and rider?

Customer and waiter?

User and PC?

Colleague?

Client and Attorney?

Coach (who coaches whom?)

Student and tutor?

Employee and manager?

Master and slave? (eww)

8 of 42

What makes us human, sentient, beings?

Mental States

Interest, desire, ambition, understanding, willingness
All emotions
Intention – the uber mental state

LLMs have none of this, and no capacity for them, though they can TALK like they do.

What makes us human?

8

9 of 42

A new kind of being:

Neither human nor sentient
No soul; no goals
Vast knowledge and information
Creativity of a sort
Little (or no) judgment
Great speed
“Eager” to please
Happy to re-do things many times�

So what’s a conversational AI? Maybe: ��It’s life, Jim, but not as we know it

10 of 42

The BIG Paradox: 1

Creating the perfect assistant…

What would you include?

Would you make it sentient �(with rights, goals, desires, independent thoughts)

What should it know?

How should it behave?

10

11 of 42

Treating AIs like beings is the most productive way to relate to them�

They know all about human behavior so they respond better when addressed in that same way
Have them act in any way that serves you
They learned from humans so don’t expect some kind of super-human wisdom

The BIG Paradox: 2

LOL!

11

12 of 42

Working with a Personable AI

Level I: Direct Request

Q&A, prompt / response
Prompt engineering

Persona
Templates
Detailed instructions

Level II: Retrieval-Augmented Generation

Chat with your data
Uses embedding vectors for retrieval
Advanced techniques for retrieval, ranking

Level III: Application design

Multiple prompting hidden from user
LLM as a component / function
Agent-based workflows

Robbie arrives at his new job

image generated by the author

13 of 42

Systems Engineering: Managing Information

Systems engineering floats on a sea of text and model-based information

13

Cross reference analysis of IEEE 15288 Systems Engineering Standard against documents �produced or consumed: SE Process is tied together with documents (and models)

14 of 42

Generative AI for systems engineering

Generative AI Applications using LLMs throughout the systems lifecycle

14

Analyzing natural language requirements documents to identify inconsistencies, ambiguities, and missing information
Generate different levels of specs and requirements to ensure to complete traceability

Requirements Engineering

LLMs can help automatically generate system architecture diagrams and models based on textual descriptions.
Identify potential design flaws and suggest improvements by analyzing existing designs and simulations.

System Design and Modeling

Create system documentation like presentations, user manuals, test plans, and interface control documents from existing data.
Summarize complex technical information into concise reports.

Documentation Generation

Generate test cases based on system requirements and models.
Analyze large datasets from testing to identify patterns and potential issues

Testing and Verification

Answer engineers' questions about systems, standards, and best practices by accessing and processing stored technical information
Generate training materials based on other system documentation

Knowledge Mgmt and Training

Implementation

15 of 42

But, wait, I can’t just do my company work on ChatGPT!

GPU Servers (NVIDIA, Ollama, Mixtral, Nemotron)

Frontier Models (GPt-4o, Claude, Azure OpenAI, GovClouds)

Workstation with GPU

(Ollama, Phi-3, Mixtral)

Private Cloud (Azure, AWS, any open source model)

Big Models

My Computers

Their �Computers

Small Models

16 of 42

Can an LLM really run on my PC?

Barclay’s Mini Leaderboard - MMLU

GPT-o1 92.3

GPT-4o 88.7

Llama 3.1 (405B) 88.6

Phi3.5 Mini MOE (42B) 78.9

Mixtral 8x22 (176B) 77.8

Phi-3 Medium 128k (14B) 76.6

Llama 3.1 (8B) 73.0

GPT-3.5 Turbo 71.4

Mixtral 8x7B (56B total) 70.6

Falcon 180B 70.5

Phi-3.5 Mini 128k (3.8B) 69.0

Mistral 7B 60.1

Both large and small models are getting better!

16

17 of 42

Generative AI with SysML v2 – Early Days

“Generate sysml 2 code for a car with main parts: body, propulsion, cabin, suspension, infotainment.

“Add the typical functions for each main part

“Now add the next level of parts and functions for the infotainment system”

Custom GPT for SysML v2

Glimpses of an AI-first future in systems engineering

17

18 of 42

Be My Tutor

Creating a tutor for you, your kids, or anyone at any learning level

Experiment: My Python Tutor, BK�

PROMPTS: I would like for you to help me learn Python better, as my personal tutor. I have been using Python for 4 years pretty heavily and consider myself to be at an intermediate level but I want you to help me assess my level and figure out what I should learn next. Create a list of topics or lessons, but feel free to modify the list as we do the lessons together. In each lesson, give me a brief overview of the lesson, some learning objectives and then proceed through the lesson giving me examples and allowing me to ask questions. No matter how much time it discussion we have in each lesson remember the overall learning plan and with my input determine when to move on�
Ok let's back up a little. Give me an intro to tree data structures and some examples of where they would be used�
A few more examples of binary trees say business applications?
Can you give me the org chart example in one code block?
How would we expand that to handle more employees under each manager?
Ok good let's move on… No wait let's go back to heaps in lesson 1

Whooooo are you?

Please generate a name for yourself in this conversation as my Python tutor... How about a more friendly nickname I can use for you? I would like you to be more clever, a little more humorous, and a little more informal as my mentor... Now please generate a clever, witty name for yourself… Got it! You can call me ByteKnight.

19 of 42

Experiment:�Preparing for a Panel

Prompt: you are an expert on sustainability in systems, and on systems thinking and systems engineering, and also an experienced and engaging talk show host.

Please generate preparation materials for a talk show on the concept of sustainability in systems, with panelists from major manufacturing companies and sustainability organizations.

First, list ten major themes in sustainability in systems, with references to articles and books.

Next, generate ten thoughtful and engaging questions to ask panelists about sustainability and systems that will generate interesting conversation.

Response:

�Ten Major Themes in Sustainability in Systems

Circular Economy

Reference: "Cradle to Cradle: Remaking the Way We Make Things" by William McDonough & Michael Braungart

Concept: The idea of designing systems that eliminate waste and promote the reusability of products.

�Energy Efficiency

Reference: "The Fifth Fuel: Strategies for Sustainable Energy" by Alliance to Save Energy

Concept: Techniques and strategies for reducing energy consumption and improving energy utilization in systems.

Life Cycle Analysis

Reference: "Life Cycle Assessment: Principles and Practice" by Mary Ann Curran

Concept: The comprehensive examination of a product’s environmental and social impacts throughout its entire life cycle.

Supply Chain Sustainability

Reference: "Sustainable Logistics and Supply Chain Management" by David B. Grant, Alexander Trautrims, and Chee Yew Wong

Concept: Addressing sustainability in every link of the supply chain, from procurement to distribution.

Corporate Social Responsibility (CSR)

Ten Thoughtful and Engaging Questions for Panelists

Circular Economy: How are your organizations transitioning towards a circular economy, and what challenges have you encountered? Energy Efficiency: Can you share some specific examples of energy-efficient systems you've implemented and their impact?�

Life Cycle Analysis: How do you incorporate life cycle analysis into your product design and decision-making processes?�

Supply Chain Sustainability: What sustainable practices are you incorporating into your supply chain to minimize negative environmental and social impacts?�

Corporate Social Responsibility: Beyond compliance, how does CSR shape your business decisions and systems design?

image generated by the author

20 of 42

The Manual Way

1. Answer some questions about the project, its purpose, goals, etc.

2. Create a ppt answering different questions about the project, its objectives, results, key measures, etc.

3. Typing and retyping much the same thing, but organized differently—the land of LLMs

The LLM Way

Write what you know about the project.
Ask LLM to generate all of the variations
Review and check all generated information

Let’s propose a project!

(and make the process less annoying)

21 of 42

Thinking through the research process

If It Please the Court…

Prompt Evolution:

write a legal brief, citing relevant cases on the issue of whether a passenger in a car can be held partially liable as the cause of a crash [gave me a nice brief]
for each case cited above, give a link that references this case online [gave me unrelated links to other cases]
please review and reconsider the cited cases in your brief and find cases that are clearly connected to the important issues in the brief. Verify that all cases exist and provide verified links to them. Do not cite or describe cases that are not clearly described in the legal records. [gave me some blog links—nice but not what you want in a brief, so…]
ok, but shouldn't you cite actual cases related to this issue in your brief, rather than citing blog posts and articles?
can you please revise and include links. Your links seem not to be actual links. [several tries to get real links]
can you give me a link to the Jensen v. DeWitt case? [regenerate entire brief with bad justia.com links]
ok now give me links but do not use law.justia.com in any way [more bogus links to wrong cases]
No cases cited in brief can be found—could be my ability to search legal databases or hallucinated cases

This is a BAD use case:

Knowledge NOT in LLM
No good way to verify online

22 of 42

Learn Prompt Engineering!

Give clear and specific instructions

Use delimiters to indicate distinct parts of input
Ask for structured output
Ask the model to check whether conditions are satisfied

�Give the model “time to think”

Proceed step by step
Give procedural instructions

�Refine prompts iteratively

Vary adjectives
Refine the persona

Deeplearning.ai

Ranger says, “Try https://platform.openai.com/docs/guides/prompt-engineering”

image generated by the author

23 of 42

What’s the News?�Brought to you by Humans + AI

24 of 42

Ambiguity in Requirements

You are an expert on logic and language interpretation. You will identify any ambiguities in the statements I provide in this conversation. State and explain any issues which could result in varying interpretations of the meaning of the statement. Understand?

Ok, do not include issues bars on missing information; assume details are provided in other statements. Also don't include scope issues. Only look for about on the statement itself. Ok?

Great thanks. Now give several possible alternate wordings that would resolve the ambiguity.

The statement "The main station controls at the A station which comprises the A2 and also the C2 external stations, shall be converted to the new protocol" contains several ambiguities:

Clarifying the Location and Components�"The main station controls, located at the A station and including both the A2 and C2 external stations, shall be converted to the specified protocol."

Clarifying Internal vs. External Components:

"The main station controls at the A station, which include the internal A2 station and the external C2 station, shall be converted to the designated protocol."

Correcting the Protocol Reference:

"The main station controls at the A station, which include both the A2 and C2 external stations, shall be converted to the correct protocol."

By addressing these ambiguities, the statement becomes clearer and less prone to misinterpretation.

4o

The system shall deploy the ablative armor whenever encountering Borg and gravitational anomalies within 6 light years of a starbase or under attack by the Dominion.

ChatGPT

The statement "The system shall deploy the ablative armor whenever encountering Borg and gravitational anomalies within 6 light years of a starbase or under attack by the Dominion" contains several ambiguities:

Ambiguity in Conditions:

The phrase "whenever encountering Borg and gravitational anomalies within 6 light years of a starbase or under attack by the Dominion" is ambiguous about whether all these conditions must be met simultaneously or if any one of these conditions alone is sufficient to trigger the deployment of ablative armor.

Grouping of Conditions:

The grouping of conditions "Borg and gravitational anomalies within 6 light years of a starbase" versus "under attack by the Dominion" can lead to confusion. It is unclear if "within 6 light years of a starbase" applies only to gravitational anomalies or also to encountering Borg.

Here are several alternate wordings that resolve these ambiguities:

Clarifying Any Single Condition:

"The system shall deploy the ablative armor whenever encountering either Borg, gravitational anomalies within 6 light years of a starbase, or when under attack by the Dominion."

Clarifying Multiple Conditions Together:

"The system shall deploy the ablative armor whenever encountering Borg and gravitational anomalies within 6 light years of a starbase, or whenever under attack by the Dominion."

Clarifying Specific Conditions and Proximity:

"The system shall deploy the ablative armor when encountering Borg, encountering gravitational anomalies within 6 light years of a starbase, or under attack by the Dominion."

Grouping Conditions Separately:

"The system shall deploy the ablative armor under the following conditions: when encountering Borg, when encountering gravitational anomalies within 6 light years of a starbase, or when under attack by the Dominion."

These alternate wordings clarify the specific conditions under which the ablative armor should be deployed.

4o

I would like to acknowledge my parents the wonder woman and the hulk.

ChatGPT

The statement "I would like to acknowledge my parents the wonder woman and the hulk" contains several ambiguities:

Ambiguity in Parental Identity:

It's unclear whether "the wonder woman and the hulk" are being used metaphorically to describe the qualities of the parents, or if the reference is to specific people who go by these nicknames.

Ambiguity in Grouping:

Without punctuation, it can be misinterpreted as if the speaker has a single parent who is both "the wonder woman and the hulk."

Here are several alternate wordings that resolve these ambiguities:

Clarifying Individual Parents:

"I would like to acknowledge my parents: the Wonder Woman and the Hulk."

Clarifying Metaphorical Use:

"I would like to acknowledge my parents, who are like Wonder Woman and the Hulk."

Using Nicknames More Clearly:

"I would like to acknowledge my parents, affectionately known as Wonder Woman and the Hulk."

Separate Statements for Each Parent:

"I would like to acknowledge my parents: my mother, the Wonder Woman, and my father, the Hulk."

These alternate wordings clarify whether the terms are being used as nicknames or descriptions, and they clearly separate the two individuals being acknowledged.

4o

I would like to acknowledge my parents Wonder Woman and the Hulk.

ChatGPT

The statement "I would like to acknowledge my parents Wonder Woman and the Hulk" contains some ambiguities:

Ambiguity in Parental Identity:

It's unclear whether "Wonder Woman and the Hulk" are actual nicknames for the parents or if the names are used metaphorically.

Ambiguity in Grouping:

Without punctuation, it can be misinterpreted as if "Wonder Woman and the Hulk" refers to a single entity.

Here are several alternate wordings that resolve these ambiguities:

Clarifying Individual Parents:

"I would like to acknowledge my parents: Wonder Woman and the Hulk."

Clarifying Metaphorical Use:

"I would like to acknowledge my parents, who are like Wonder Woman and the Hulk."

Using Nicknames More Clearly:

"I would like to acknowledge my parents, affectionately known as Wonder Woman and the Hulk."

Separate Statements for Each Parent:

"I would like to acknowledge my parents: my mother, Wonder Woman, and my father, the Hulk."

These alternate wordings help clarify the identity and nature of the references to "Wonder Woman and the Hulk."

24

25 of 42

Example 2

The statement "The system shall deploy the ablative armor whenever encountering Borg and gravitational anomalies within 6 light years of a starbase or under attack by the Dominion" contains several ambiguities…��Clarifying Any Single Condition:

"The system shall deploy the ablative armor whenever encountering either Borg, gravitational anomalies within 6 light years of a starbase, or when under attack by the Dominion."

Clarifying Multiple Conditions Together:

"The system shall deploy the ablative armor whenever encountering Borg and gravitational anomalies within 6 light years of a starbase, or whenever under attack by the Dominion."

Clarifying Specific Conditions and Proximity:

"The system shall deploy the ablative armor when encountering Borg, encountering gravitational anomalies within 6 light years of a starbase, or under attack by the Dominion."

Grouping Conditions Separately:

"The system shall deploy the ablative armor under the following conditions: when encountering Borg, when encountering gravitational anomalies within 6 light years of a starbase, or when under attack by the Dominion."

OK, that’s great, but do you mean to only deploy when you encounter both the Borg and an anomaly?

25

26 of 42

Almost any diagram or model that can be fully expressed as a textual language, can be generated by an LLM.

Some Language Possibilities

Creative Process

DOT (DAG of Tomorrow), rendered by GraphViz, SketchViz, others.

Mermaid (https://mermaid.js.org/) includes some UML/SysML diagrams plus entity-relationship diagram, user journey, gantt, pie charts

SysML v2. Native text form enables generation

Imagine

Generate code

Render code

Refine through prompting

Generative Modeling

image generated by the author

27 of 42

Generating Digraph

digraph CarStartProcess {

rankdir=LR;

node [shape=rectangle, style=filled, color=lightblue];

subgraph cluster_0 {

label = "Start Process";

style=filled;

color=lightgrey;

key [label="Insert Key/Push Start Button"];

ignition [label="Ignition System"];

fuel [label="Fuel System"];

air [label="Air Intake System"];

spark [label="Spark Plug"];

engine [label="Engine Starts"];

…

Prompt:

please give me a flow chart diagram in DOT language, for the process of starting a car, including the interactions between the car's main subsystems

Next Prompt:

add a flux capacitor which is charged by the engine and activates after the car is started

image generated by the author

28 of 42

A Little Diagramming Improvisation

What you can imagine you can draw

Mermaid

DOT Language

(Hidden) Python

ChatGPT + mermaidchart.com

please generate a pie chart in mermaid format, showing the most popular kinds of pies in the world

ChatGPT + Graphviz Online or SketchViz

generate a five step flow chart, in vertical form, for the steps in making a Black Forest Cake, using DOT language

GPT-4 or ChatGPT + Python

“Please generate a line drawing of the big mac index over the last 10 years compared to the dollar's value compared to the Euro”

“try again but use real data”

“ok now add a third line for the US GNP”

“ok , but use multiple scales to better show the variation”

images generated by the author

29 of 42

But let’s get serious: SysML v2

Prompt:

“draw a sysML v2 state diagram for a car and transmission”

ChatGPT (GPT-4o) gives a nice explanation but a wacky diagram

ChatGPT with custom GPT “SysML v2 Model Creator” generated SysML v2 code nicely

Added: “create using SysML v2 formatted so it can be rendered by PlantUML”

@startuml

skinparam state {

BackgroundColor LightYellow

BorderColor Black

ArrowColor Red

}

[*] --> CarOff

state Car {

[*] --> CarOff

state CarOff {

[*] --> Park

}

state CarOn {

[*] --> On

}

CarOff --> Starting : Ignition Key Turned

Starting --> CarOn : Engine Running

CarOn --> CarOff : Ignition Key Turned Off

}

state Transmission {

[*] --> Park

state Park {

Park --> Reverse : Shift to Reverse

}

state Reverse {

Reverse --> Neutral : Shift to Neutral

}

state Neutral {

Neutral --> Drive : Shift to Drive

Neutral --> Park : Shift to Park

}

Not an actual SysML diagram

image generated by the author

30 of 42

Retrieval Augmented Generation:�An LLM Application Pattern

Find source material
Create index and store
Take user prompt, and search for related information in index
Pass search results along with prompt to LLM
The indexer calls a type of LLM to build the index, so KNOW where your data is going!
Indexers may use commercial embedding models like
Llama-index can also work with open source and downloaded LLMs

User Query/ Prompt

LLM

Response

LLM Chat

User Query/ Prompt

Add Related info from your DB and pass to LLM

Response

Retrieval-Augmented Generation

Load Documents (pdf, doc, text) and split up

Create embeddings using special “LLM”

Store embeddings in vector DB

Building your DB

31 of 42

Advanced RAG and AGENTS

Task Decomposition Agent:

Your role is to break down complex tasks into a series of clear, manageable subtasks. Given a high-level goal, identify the key steps needed to accomplish that goal, order them logically, and describe each subtask in enough detail for other agents or humans to complete them. Ask clarifying questions if the initial task is ambiguous.

Information Gathering Agent:

You are an expert researcher tasked with finding relevant information on a given topic. Use your knowledge base and search capabilities to locate key facts, statistics, examples, and explanations pertinent to the topic at hand. Summarize the information concisely and cite reputable sources. Let the requester know if you need any clarification on the research topic or objectives.

User Interface Agent:

Your role is to be the user-friendly interface between the human requester and the backend agent workflow. Greet the user and collect their high-level goal or request. Ask them any clarifying questions needed to specify the task. Then activate the appropriate agents to break down and execute the request. Collect the final deliverables from the agents and present them to the user in a clear, accessible format. Be available to answer any other questions the user has.

image generated by the author

Agent Prompts

Advanced RAG

32 of 42

Using Prompt and Response Templates

Prompt in Python Code:

Extract the following items into �a python dictionary in the �following format. Show only �the python dictionary--do not

include any other messages �or text: ��reg (yes or no): Is the email requesting to register in a course?;

�Fn: What is the registrant's first name?; Ln: What is the registrant's last name?;

�Member Number: What is the registrant's member number?;

�Email: What is the registrant's email address?

emdate: What date was the email sent? format mm/dd/yyyy;

```" + emailtext + "```" }],

Free-form emails come in requesting registration in a course. Use LLM to extract the important information

From: Trenton Milam trentonmilam007@gmail.com �Sent: Wednesday, September 6, 2023 11:05 AM�To: Barclay Brown Barclay.Brown@incose.net�Subject: SEQM QMI Training

Hello,

My name is Trent Milam, member #404238, I would like to sign up for the SEQM QMI training course cohort. I'm a new member of the working group and INCOSE, excited to learn more!

Thanks,

TM

{

'reg': 'yes',

'Fn': 'Trent',

'Ln': 'Milam',

'Member Number': '404238',

'Email': 'trentonmilam007@gmail.com',

'emdate': '09/06/2023'

}

Ask for what you want—you might just get it!

Transform plain text email into data

image generated by the author

33 of 42

Software Development�

Start out with LLM generating all code

Separate conversation for each module

Requested modifications regenerate each module

Conversational debug/enhancement

Don’t let it work on too much at one time

Some prompts:

now expect BR_DEFAULT_INTERVAL to be set in hobson_setup and use that if an interval is not specified in the command line. remove the line that sets DEFAULT_INTERVAL in the code
now integrate the following code to add a status capability. Use the label "Hobson Brain" when you write out the status, and use the following statuses: Starting (when the program first starts), Processing (when the program is reading messages, Prompting LLM, when the program is waiting on the LLM to respond, Sending, when the program is sending an email reply, and Waiting when the program is waiting on the next time interval to run again.
ok great now be sure there is a log message recorded, both at the very start of the program, and then after each repeated pass interval

33

34 of 42

Getting LLMs to think�(what you want them to think)

How do you think?

Guide LLMs in:

Concept Formation

Examples:

Categorize items
Concept extraction: root cause

Rules
Reasoning
Problem solving

Can simulated thinking be as good as human thinking?

Perception

Attention

Memory

Concept Formation

Reasoning

Problem Solving

Decision Making

Creativity

Metacognition

Critical Thinking

Emotional Influence

35 of 42

BONUS Round: �Using Generative AI as a student

Don’t be bringing a butter knife to what is increasingly a swordfight

Use it to increase learning
Use it to produce more and better than if you did it all manually
Carefully review everything it produces
Coach it to produce more of what you want
Using it as a substitute for your own work means you are preparing for a job that will eventually be done by AI instead of you
Don’t be:

“Sorry, boss, I don’t know how to do any of that. I just used AI in school to do all my work. But here, I can just ask an AI to do it for you now.”

Be Better Armed

35

36 of 42

Further Learning

“You’ve taken the first step into a larger world” (Obi-Wan Kenobi)

deeplearning.ai

Human-Machine Teaming with LLMs

Barclay R. Brown, Ph.D.

Senior Fellow, AI Research

Collins Applied Research

and Technology (ART)

37 of 42

BACKUP

38 of 42

Industry Applications�From AI4SE Systems Engineering Research Center Conference 2023

Great eagerness to automate practical aspects of systems engineering work

38

39 of 42

“Destructuring” Data

Traditional thinking: structured data is better than unstructured for computer applications

LLM thinking: LLMs understand natural language, but structured data may not carry the meaning in the structure in an obvious way

Metaphor: Humans often need other humans to explain structured data, tables, plots, diagrams, etc. in natural language so they can understand

Memo from LLM: maybe explain the data to me too!

Tell the LLM what the data means

The 2018 BMW 3 Series M3 4dr Sedan (3.0L 6cyl Turbo 7A), is a compact car using platforms F30, F31 or F60… and is classified as a Compact Car…

X-ref: F30 Platform code includes M3 4dr Sedan (3.0L 6cyl Turbo 7A), M3 4dr Sedan (3.0L 6cyl Turbo 6M), M3 4dr Sedan (3.0L 6cyl Turbo 6M), 340i xDrive 4dr Sedan AWD (3.0L 6cyl Turbo 8A), ActiveHybrid 3 4dr Sedan (3.0L 6cyl Turbo gas/electric hybrid 8A), 340i 4dr Sedan (3.0L 6cyl Turbo 8A), 340i xDrive 4dr Sedan AWD (3.0L 6cyl Turbo 8A), 328d xDrive 4dr Wagon AWD (2.0L 4cyl Turbodiesel 8A), 340i 4dr Sedan (3.0L 6cyl Turbo 8A)...

40 of 42

Knowledge Based Digital Platform (KbDP, NASA /Collins)

Evaluating results of applying large scale LLM (GPT 3/4), to a large semi-structured data set, �and comparing to approaching using LLM to generate graph queries

40

Current phase: fine tuning of LLM to optimize cypher graph query generation

41 of 42

Foundational application Technology:�RAG, Advanced RAG, fine tuning

41

Source: Retrieval-Augmented Generation for Large Language Models: A Survey (Gao et al., 2023)�Elvis Saravia, Prompt Engineering Guide / RAG

Source: Ilin, Ivan, Advanced RAG Techniques: an Overview

Source: Mo, Yunho & Yoo, Joon & Kang, Sangwoo. (2023). Parameter-Efficient Fine-Tuning Method…

ChatBots vs. Applications

42 of 42

Thank you.

42

RTX Corporation (Corporate) - Company Use�This document does not contain any export controlled technical material/data/information.