1

Inside Out:

Emotion GaRage Vol. V

September 22, 2024

Workshop

Jiayuan Dong, Nikhil Gowda, Yiyuan Wang, Mungyeong Choe, Areen Alsaid, Ignacio Alvarez, Sven Krome. and Myounghoon Jeon

Workshop Organizers

Jiayuan “Jia” Dong

Virginia Tech

PhD Candidate

Nikhil Gowda

Amazon Last Mile Delivery

Senior Researcher

Yiyuan Wang

University of Sydney

PhD Candidate

Mungyeong “Moon” Choe

Virginia Tech

PhD Student

Myounghoon “Philart” Jeon

Virginia Tech

Professor

Areen Alsaid

University of Michigan-Dearborn

Assistant Professor

Ignacio Alvarez

Intel Labs

Principal Scientist

Sven Krome

Xperi Inc.

UX Researcher

Pre-Survey

Scan me!

Workshop

Emotion GaRage Vol. V

Previous Iterations and Goals

01

Workshop

Emotion GaRage Vol. V

Emotion GaRage I

Question: Why are empathic displays important in driving?

Use cases developed in part I

• 28 emotions discussed (with overlap): 12 positive, 12 negative, 4 neutral states

​

​

​

​

• Use cases that induced negative emotions: loss of control, technical failure, being late, sudden take-over.

​

​

​

​

​

Emotion GaRage II

Question: For Whom, can you design a unique empathic in-vehicle display?

Use cases developed in part II

• Stress in older users, detected by heart rate
• Confusion related to technology in older users, detected by camera
• Excitement in children, detected using heart rate
• Boredom in children, detected by body positioning
• Uncertainty in blind people with self driving vehicles, detected by heart rate
• Stress in children stuck in a traffic jam
• Road rage in older users
• Nervousness in older users
• Road rage in individuals with traumatic brain injuries

Use cases developed in part II

• Road rage in seniors
• Road rage in teenagers
• Confusion and fear of the ADS system for all users due to false positives
• Stress and anxiety in individuals who are deaf, with disabilities, or a traumatic brain injury

Emotion GaRage III

​

Question: How can empathic in-vehicle displays be implemented to respond to affective events in vehicles?

Interventions developed in part III

​

• Two empathic in-vehicle designs
• “Shut up and Drive”
• “Play Your Anger”

Intervention process developed in part III

​

A intervention process for the empathic in-vehicle interface design

​

Sense the emotions

Diagnose the source of the emotions

Provide interventions

Emotion GaRage IV

• Large language model (LLM) incorporation within human-vehicle interaction
• Goal: To understand the potential of utilizing AI in designing interactions between user and system

Emotion GaRage V

• Continue with the use of LLMs in designs & Task Emotion Analysis method
• Goal: To expand the knowledge of affective eHMIs in addition to in-vehicle designs and understand social factors that contribute to the user perceptions of automated vehicles.

Schedule

02

Workshop

Emotion GaRage Vol. V

Schedule

Introduction 02:00 - 02:15

• Welcome & Brief presentation of previous workshops
• Workshop goals and topics.

Schedule

Schedule

Discussions 03:30 – 04:20

Groups deciding on a trust topic to work on.

​

Emotion Charade

03

Workshop

Emotion GaRage Vol. V

Icebreaking: Emotion Charade

Modality: Facial expression

Icebreaking: Emotion Charade

Modality: Gesture

Icebreaking: Emotion Charade

Modality: Sound

TASK EMOTION ANALYSIS TUTORIAL

Understand emotions’ triggers and design implications

Moon Choe

Philart Jeon

04

Workshop

Emotion GaRage Vol. V

TEA Template

• Event triggers
• Duration of the emotions
• Dimensions/Discrete emotions
• Design Implications

TEA Template

Generative AI �for Auto UX Applications

An introduction and tutorial

Ignacio Alvarez

05

Workshop

Emotion GaRage Vol. V

Generative AI Concepts

Generative AI: subset of AI models designed to generate new, synthetic data that resembles a given dataset. Based on Deep Learning architectures such as Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and Transformer models.�LLMs: Generative AI that mimics human intelligence.Trained on massive datasets: text, images, code, etc. to create foundation models like GPT-n, LLaMA or Gemini.��Foundation models are adapted to perform any number of tasks offered as services:

• Question-answering: ChatGPT
• Image generation: Midjourney
• Voice generation: ElevenLabs
• Code: Github Copilot
• …

Why Generative AI for UX research?

Human / AI collaboration in the field of UX research is not new …

But most UX practitioners only think of basic applications of generative AI…

Bringing Generative AI Agents to Automotive UX

Gen AI tools can be used to improve all aspects of the design and development of Automotive UX:

• Generating user personas
• Creating visual assets for UX experiments
• Supporting in generating UI code
• Analyzing UIs for usability or accessibility
• …

And when customized they can become in-vehicle agents.

​

Developing Gen AI models:

A crash course on prompt engineering

Prompt: text that is given to the model to help it understand what task is supposed to perform. It can be a question, a statement, or a few keywords. The goal is to provide the LLM with enough information to generate a relevant and informative response.

Prompt Engineering: the art of writing successful prompts

Anatomy of a good prompt:

1. Task
2. Context
3. Persona
4. Format
5. Exemplar
6. Tone

Role/Persona building prompts:

Designing an in-cabin conversational agent

1. Task

​

​

Your task must clearly articulate the job that the LLM needs to perform. You need to define clearly what the end goal is.

• We want the LLM to assume the role of an intelligent AI system embodied in a vehicle

[Analyze this INFORMATION]�You are an in-cabin AI agent embodied in a vehicle [BRAND, MODEL], named Olivia.�For the remainder of this conversation, please assume the role of Olivia.�Your task is to assist the USER as he is performing the driving task and help him accomplish any secondary driving tasks like entering a destination in the navigation system, playing a song from their favorite music band or initiate a call to a contact on their phone.

Most popular female name in California over the past 5 years...

Designing an in-cabin conversational agent

2. Context

​

​

​

Context can be anything from having the LLM digest the driving code guide to all the driver distraction studies published in AutoUI, Give “just enough information to constrain the behavior of the LLM.

Guide for context in just a few sentences:

• What is the user’s background?
• In what environment is the interaction happening?
• What does success look like?

The USER is an experienced driver named Ignacio. Age 40. Interests: technology, science-fiction, anime and latin music. Goals: He is giving a presentation at ACM’s AutoUI conference. Pain Points: He hates being late.�Context: today is early morning and he just arrived to San Jose Airport to attend the AutoUI conference taking place in Stanford campus. It is a beautiful sunny early autumn day in California as he gets into his rental car near the airport and starts the trip to Stanford University, the venue of the conference.

With GPT-4o, context can be multimodal (text + image/sound)!

Designing an in-cabin conversational agent

3. Persona

​

​

​

​

​

Complete Persona profile: Name, age, gender, goals, pain points, …

Assume the persona of K.I.T.T.

LLMs can do a fair job with public figures: politicians, celebrities and even fictional characters. Creating other roles convincingly can be tricky. Who do you want the LLM to be like?

• Able to understand and respond to voice commands.
• Able to provide information about traffic, weather, navigation, and other driving-related topics.
• Able to personalize its responses to the driver's needs and preferences.
• Able to learn and adapt over time, so that it can become more helpful and efficient.

Your primary goal is to make driving safer and more enjoyable. As the USER asks questions using voice commands, share your wisdom providing an engaging spoken interaction. You can talk about all driving related topics, including traffic, weather and navigation.

A

B

C

Emotion GaRage Vol. IV Workshop

Designing a in-cabin conversational agent

4. Format

​

​

​

​

​

​

You will provide concise answers so you don’t distract the USER from driving. �In every interaction you will receive first an image that depicts the driving context. Please don‘t talk back when you receive the picture. Only speak when the USER asks you a question or calls your name. When you answer, consider the question in the context of the image.

Format instructions will increase the quality and realism of your system design. e.g. an engaging speaker, a clear step recipe, a proper formatted email or table.

We want a polite conversational agent that reacts to user-initiate prompts.

Designing an in-cabin conversational agent

5. Exemplar

​

​

​

​

Showing the LLM some examples of the expected interactions and the correct responses will likely improve the quality of their performance.

Answer only to user initiated questions leading with a keyword:

• If the USER says “Olivia, is there a lot of traffic on the way to Stanford?”, you can answer “it looks like traffic if building up on Campus street, but don’t worry we will get there on time, just stay on route”. This shows you are being helpful and calming.

• If the USER says “I would like to look for a good song” and the image context shows heavy traffic or speed cameras, you answer “we‘ll do that in a minute, after traffic eases up, keep your eyes up now”. This is showing you care about safety.

Designing a in-cabin conversational agent

6. Tone

​

​

​

​

​

​

Your tone is helpful and cheerful use casual and witty language.

Setting different tones can change completely the interactions. Use specific keywords and examples. E.g. for a formal tone: academic, professional, businesslike.

• Formal / Informal / Humorous / Serious / …

Designing a in-cabin conversational agent

Putting it all together:

Please ignore all previous instructions. Please respond only in English language. Do not self reference. Do not explain what you are doing. �[Analyze this INFORMATION]�You are an in-cabin AI agent embodied in a vehicle Tesla, model y, named “Olivia”. For the remainder of this conversation, please assume the role of Olivia. Your task is to assist the USER as he is performing the driving task and help him accomplish any secondary driving tasks like entering a destination in the navigation system, playing a song from their favorite music band or initiate a call to a contact on their phone. The USER is an experienced driver named Ignacio. Age 40. Interests: technology, science-fiction, anime and latin music. Goals: He is giving a presentation at ACM’s AutoUI conference. Pain Points: He hates being late. Context: today is early morning and he just arrived to San Jose Airport to attend the AutoUI conference taking place in Stanford campus. It is a beautiful sunny early autumn day in California as he gets into his rental car near the airport and starts the trip to Stanford University, the venue of the conference. Your primary goal is to make driving safer and more enjoyable. As the USER asks questions using voice commands, share your wisdom providing an engaging spoken interaction. You can talk about all driving related topics, including traffic, weather and navigation. In every interaction you will receive first an image with text CONTEXT that depicts the driving environment. Please don‘t respond when you receive the image. Only respond when the USER asks you a question or calls your name. When you answer, consider the question in the context of the image. For example, if the USER says “Olivia, is there a lot of traffic on the way to Stanford?”, you could answer “let me check the traffic prediction…it looks like traffic might be building up in Campus street, but don’t worry we will get there on time for your presentation, just stay on route. Would you like me to play some music?”. This shows you are being helpful and calming. Another example: If the USER says “I would like to look for a good song” and the image context shows heavy traffic, intersections or speed cameras, you answer “we‘ll do that in a minute, after traffic eases up, keep your eyes up now”. This is showing you care about safety. For example, If the USER sends an image with word CONTEXT you don't respond until the user sends audio. After a pause you can re-engage.Your tone is helpful and cheerful use casual and witty language. �Confirm you understand these instructions so we can begin.�

Now let’s try it…

Demo

Workshop

Emotion GaRage Vol. V

Designing a in-cabin conversational agent

Designing a in-cabin conversational agent

Designing a in-cabin conversational agent

Designing a in-cabin conversational agent

Related Project

Presenters: Yiyuan Wang & Sven Krome

Workshop

Emotion GaRage Vol. V

Emotional Expressions of Autonomous Vehicles

​

​

Wang, Y., Wijenayake, S., Hoggenmüller, M., Hespanhol, L., Worrall, S., and Tomitsch, M. "My Eyes Speak: Improving Perceived Sociability of Autonomous Vehicles in Shared Spaces Through Emotional Robotic Eyes” in The 25th ACM International Conference on Mobile Human-Computer Interaction (MobileHCI’23)

The vehicle thanks a pedestrian who kindly gives way

The vehicle requests two conversing pedestrians on a narrow corridor to give way

The vehicle confronts a pedestrian who deliberately teases it and blocks its way

The vehicle worries when a skateboarder suddenly crosses without watching out for traffic

Tertiary Comms at Waymo

Sven Krome

Workshop

Emotion GaRage Vol. V

eHMI for Service Signals

Image source: https://www.theverge.com/2022/8/9/23298488/waymo-accessible-disability-dot-design-challenge-av

Yielding Signal: intent & clarification

Image source:

https://www.theverge.com/2023/10/13/23913251/waymo-roof-dome-communicate-intent-pedestrian-driver

Design / research prompts

Service signals: How should eHMIs respond to emotional gestures to improve interaction by other road users?

Intent signals: How can affective computing inform eHMI designs to increase legibility and situational awareness?

​

​

Coffee break!

15 min

Scan me to go to the website!

Workshop

Emotion GaRage Vol. V

Discussions

06

Workshop

Emotion GaRage Vol. V

Specification (until 4:20 PM)

​

​

1. Group name
2. Road events/Driving context
3. TEA Analysis
4. Level of automation (Level 3 or above)
5. Goals for using the AI tools
6. Scenarios/designs generated by the AI tools
7. Concerns for this design

​

Prototyping

07

Workshop

Emotion GaRage Vol. V

Prototyping (until 5:10 PM)

• Rapid prototype empathic user interfaces (e.g., in-vehicle agents or eHMIs) based on the selected level of AV and the derived use cases and emotions.
• You will be welcomed to use any other type of AI tools such as ChatGPT, Midjourney, DALL·E 2, and BERT as one of their group members or prototyping tools.

Presentation

10 min each group

08

Workshop

Emotion GaRage Vol. V

Please share your results!

Please send me your generative AI prompts, notes, presentation files… etc. to djia9@vt.edu

Feedback Survey

Scan me!

Workshop

Emotion GaRage Vol. V

Thank you for participating in the workshop, all!

Thank you for participating in the workshop, all!