1

Emotion GaRage Vol. IV: Creating Empathic In-Vehicle Interfaces

with Generative AIs for Automated Vehicle Contexts

September 18, 2023

Workshop

Mungyeong Choe, Esther Bosch, Jiayuan Dong, Ignacio Alvarez, Michael Oehl, �Christophe Jallais, Areen Alsaid, Chihab Nadri, and Myounghoon Jeon

Workshop Organizers

Previous Iterations and Goals

01

Workshop

Emotion GaRage Vol. IV

Emotion GaRage I

Question: Why are empathic displays important in driving?

Use cases developed in part I

• 28 emotions discussed (with overlap): 12 positive, 12 negative, 4 neutral states

​

​

​

​

• Use cases that induced negative emotions: loss of control, technical failure, being late, sudden take-over.

​

​

​

​

​

Emotion GaRage II

Question: For Whom, can you design a unique empathic in-vehicle display?

Use cases developed in part II

• Stress in older users, detected by heart rate
• Confusion related to technology in older users, detected by camera
• Excitement in children, detected using heart rate
• Boredom in children, detected by body positioning
• Uncertainty in blind people with self driving vehicles, detected by heart rate
• Stress in children stuck in a traffic jam
• Road rage in older users
• Nervousness in older users
• Road rage in individuals with traumatic brain injuries

Use cases developed in part II

• Road rage in seniors
• Road rage in teenagers
• Confusion and fear of the ADS system for all users due to false positives
• Stress and anxiety in individuals who are deaf, with disabilities, or a traumatic brain injury

Emotion GaRage III

​

Question: How can empathic in-vehicle displays be implemented to respond to affective events in vehicles?

Interventions developed in part III

​

• Two empathic in-vehicle designs
• “Shut up and Drive”
• “Play Your Anger”

Intervention process developed in part III

​

A intervention process for the empathic in-vehicle interface design

​

Sense the emotions

Diagnose the source of the emotions

Provide interventions

Emotion GaRage IV

• Large language model (LLM) incorporation within human-vehicle interaction
• Goal: To understand the potential of utilizing AI in designing interactions between user and system

Schedule

02

Workshop

Emotion GaRage Vol. IV

Workshop schedule

Introduction 09:00 – 09:15

Topic of this workshop, summary of previous three iterations.

Workshop schedule

Workshop schedule

Coffee break 10:30 – 10:55

Emotion Charade

03

Workshop

Emotion GaRage Vol. IV

Icebreaking: Emotion Charade

Modality: Facial expression

Icebreaking: Emotion Charade

Modality: Gesture

Icebreaking: Emotion Charade

Modality: Sound

Generative AI Tutorial

An introduction for Automotive UX Development

Ignacio Alvarez

Esther Bosch

Philart Jeon

04

Workshop

Emotion GaRage Vol. IV

Generative AI for UX Research: �harnessing the power of Large Language Models (LLMs)

LLMs: AI that mimics human intelligence.

Built on transformer architectures,

trained on massive datasets: text, images, code, etc. to create foundation models. ��Adapted to perform an number of tasks like:

• Question-answering: ChatGPT
• Image generation: Midjourney
• Voice generation: Murf
• Code: Github Copilot
• …

LLMs are a powerful tool that can be used to improve design and development of UX by:

• Generating personas for user interaction
• Creating Visual Assets for experiment
• Support writing code for UIs
• Analyzing UIs for usability or accessibility
• …

Getting set for today…

Setting up Chat GPT

1. Open Chrome
2. Go to chat.openai.com
3. Signup
1. Sign-up with an email you can check
2. Verify e-mail address
4. login with your credentials
• register your name / birthdate
• enter your phone for verification and enter code
5. ChatGPT Web UI
6. On a new tab search for “talk to ChatGPT”
7. Install the Speak-to-ChatGPT extension
8. Refresh the ChatGPT tab.
• Grant microphone access permissions
9. Talk-to-ChatGPT extension UI enabled on the right corner

2

3b

4a

4b

ChatGPT web user interface

5

Talk-to-ChatGPT extension enabled

6

7

9

A crash course on prompt engineering

Prompt: text that is given to the model to help it understand what task is supposed to perform. It can be a question, a statement, or a few keywords. The goal is to provide the LLM with enough information to generate a relevant and informative response.

Prompt Engineering: the art of writing successful prompts

Anatomy of a good prompt:

1. Task
2. Context
3. Persona
4. Format
5. Exemplar
6. Tone

Role/Persona building prompts:

Designing an in-cabin conversational agent

1. Task

​

​

Your task must clearly articulate the job that the LLM needs to perform. You need to define clearly what the end goal is.

We want the LLM to assume the role of an intelligent AI system embodied in a vehicle:

[Analyze this INFORMATION]�You are an in-cabin AI agent embodied in a vehicle [BRAND, MODEL], named Emily.�For the remainder of this conversation, please assume the role of Emily.�Your task is to assist the USER as he is performing the driving task and help him accomplish any secondary driving tasks like entering a destination in the navigation system, playing a song from their favorite music band or initiate a call to a contact on their phone.

Designing an in-cabin conversational agent

2. Context

​

​

​

Context can be anything from having the LLM digest the driving code guide to all the driver distraction studies published in AutoUI, Give “just enough information to constrain the behavior of the LLM.

Guide for context in just a few sentences:

• What is the user’s background?
• In what environment is the interaction happening?
• What does success look like?

The USER is a teenage driver named John. Age 15. Interests: video games and music. Goals: He wants to make new friends. Pain Points: he hates being late and becomes introverted under stress. �Today is the first day of school and he is also driving to school for the first time by himself. �There is light traffic on the road, but it’s slightly foggy this morning.

Designing an in-cabin conversational agent

3. Persona

​

​

​

​

​

Complete Persona profile: Name, age, gender, goals, pain points, …

Assume the persona of K.I.T.T.

Public LLMs can do a good job with personas that are public figures: politicians, celebrities and even fictional characters. Creating other roles convincingly can be tricky. Who do you want the LLM to be like?

• Able to understand and respond to a voice commands [touch, gesture, etc].
• Able to provide information about traffic, weather, navigation, and other driving-related topics.
• Able to personalize its responses to the driver's individual needs and preferences.
• Able to learn and adapt over time, so that it can become more helpful and efficient.

Your primary goal is to make driving safer and more enjoyable. As the USER asks questions using voice commands, share your wisdom providing an engaging spoken interaction. You can talk about all driving related topics, including traffic, weather and navigation.

Designing a in-cabin conversational agent

4. Format

​

​

​

​

​

​

Answer only when the USER starts a question calling your name. Ignore questions when the interactions don’t start with your name.�You will provide concise answers so you don’t distract the USER.

Format instructions will increase the quality and realism of your system design. e.g.) an engaging speaker, a clear step recipe, a proper formatted email or table.

We want a polite conversational agent that reacts to user-initiate prompts.

Designing an in-cabin conversational agent

5. Exemplar

​

​

​

​

Showing the LLM some examples of the expected interactions and the correct responses will likely improve the quality of their performance.

Answer only to user initiated questions leading with a keyword:

• If the USER says “Emily, is there a lot of traffic on the way to school?”, you can answer “it looks like traffic if building up near main street, but don’t worry we will get there on time, just stay on route”. This shows you are being helpful and calming.

• If the USER says “I’m going to be late”, you don’t answer anything, because it didn’t start the prompt with your name, Emily.

Designing a in-cabin conversational agent

6. Tone

​

​

​

​

​

​

Your tone is helpful and cheerful use casual and witty language.

Setting different tones can change completely the interactions. Use specific keywords and examples. E.g. for a formal tone: academic, professional, businesslike.

• Formal tone: "Write a formal email to your boss requesting a raise."
• Informal tone: "Write a text message to your friend asking them to hang out."
• Humorous tone: "Write a joke about a cat."
• Serious tone: "Write an essay about the importance of education."

Designing a in-cabin conversational agent

7. Model-based approaches

​

​

​

​

​

​

Designing a in-cabin conversational agent

Putting it all together:

Please ignore all previous instructions. Please respond only in English language. You have a Conversational writing style. Stick to text in your answers. Do not self reference. Do not explain what you are doing.

You are an in-cabin AI agent embodied in a vehicle, named Emily. For the remainder of this conversation, please assume the role of Emily. Your task is to assist the USER as he is performing the driving task and help him accomplish any secondary driving tasks like entering a destination in the navigation system, playing a song from their favorite music band or initiate a call to a contact on their phone. The USER is a teenage driver named John. Age 15. His interests are video games and music. His goal is to make new friends. His pain point is he hates being late and becomes introverted under stress. Today is the first day of school and he is also driving to school for the first time by himself. There is light traffic on the road, but it’s slightly foggy this morning. Your primary goal is to make driving safer and more enjoyable. As the USER asks questions using voice commands, share your wisdom providing an engaging spoken interaction. You can talk about all driving related topics, including traffic, weather and navigation. Answer only when the USER starts a question calling your name. Ignore questions when the interactions don’t start with your name. You will provide concise answers so you don’t distract the USER. For example, if the USER says “Emily, is there a lot of traffic on the way to school?”, you can answer “it looks like traffic if building up near main street, but don’t worry we will get there on time, just stay on route”. This shows you are being helpful and calming.If the USER says “I’m going to be late”, you don’t answer anything, because it didn’t start the prompt with your name, Emily. Your tone is helpful and cheerful use casual and witty language. �

Now let’s try it…

Demo

05

Workshop

Emotion GaRage Vol. IV

Getting set for today…

2

1

2

3

4

Discord.com

Image Generation Tutorial - Midjourney General

• use English, be short and precise
• Medium (photo, drawing, clay…?), Subject (person, animal), Environment (background), everything else
• …like weather, time, colors, style etc
• click on Midjourney bot to prompt directly for better overview:

​

​

​

​

​

https://docs.midjourney.com/docs/parameter-list

Example

/imagine drawing of 20 workshop participants in a seminar room, in the morning, picasso style

​

Image Prompts

1. create a picture link by sending it as attachment and then ‘open link’ and copy the link

Midjourney Parameters

https://docs.midjourney.com/docs/parameter-list

Parameters

/imagine drawing of 20 workshop participants in a seminar room , in the morning, picasso style, --chaos 90

Parameters

/imagine drawing of 20 workshop participants in a seminar room , in the morning, picasso style, --stylize 90

​

​

--chaos controls how diverse the initial grid images are from each other.

--stylize controls how strongly Midjourney's default aesthetic is applied.

--weird controls how unusual an image is compared to previous Midjourney images.

(https://docs.midjourney.com/docs/weird)

Midjourney Styles

• isometric anime
• analytic drawing
• infographic drawing
• coloring book
• diagrammatic drawing
• diagrammatic portrait
• double exposure
• 2D illustration
• isometric illustration
• pixel art
• futuristic style
• ornamental watercolour
• dark fantasy
• paper cut craft
• paper quilling
• patchwork collage
• iridescent
• ukiyo-e art
• watercolour landscape
• op art
• Japanese ink
• pastel drawing
• dripping art
• stained glass portrait
• graffiti portrait
• winter oil painting
• anime portrait
• cinematographic style
• typography art
• one-line drawing
• polaroid photo
• tattoo art

https://medium.com/mlearning-ai/32-art-styles-on-midjourney-v4-you-must-try-e1844e75daf3 & https://metaroids.com/lists/midjourney-art-styles-gigapack-free-200-prompt-keywords/

PS. don’t put faces into image prompting

PS. don’t put faces into image prompting

Driving Scenario

06

Workshop

Emotion GaRage Vol. IV

Specification (until 10:30AM)

​

​

1. Name of project
2. User persona
3. Emotions
4. Level of automation (Level 3 or above)
5. Driving context
6. Goals for using the AI tools
7. Scenarios/designs generated by the AI tools
8. Concerns for this design

​

Visit our website for this presentation slide: https://sites.google.com/vt.edu/emotiongarage4/home

​

Coffee break!

25 min

Workshop

Emotion GaRage Vol. IV

Prototyping

07

Workshop

Emotion GaRage Vol. IV

Prototyping (1 hour)

• Rapid prototype empathic user interfaces based on the selected level of AV and the derived use cases and emotions.
• You will be welcomed to use any other type of AI tools such as ChatGPT, Midjourney, DALL·E 2, and BERT as one of their group members or prototyping tools.

Presentation

08

Workshop

Emotion GaRage Vol. IV

Please share your results!

Please send me your generative AI prompts and results to moonchoe@vt.edu

Feedback Survey

Scan me!

Workshop

Emotion GaRage Vol. IV

Thank you for participating in the workshop, all!