1 of 36

AutoUI 2022: Workshop on Automotive Mixed Reality Applications: Transitional Interfaces, Multi-User VR, and Helmet-Mounted AR for Cyclists

Andreas Riegler, Andreas Riener, Philipp Wintersberger, Tamara von Sawitzky, Ye Eun Song

18.09.2022, Seoul, South Korea

2 of 36

Agenda

09:00 – 09:10 Agenda, Round of Introductions
09:10 – 09:30 Introduction to the Topics

Automated Driving (AD)
Mixed Reality (MR)
Transitional Interfaces
Multi-User VR
AR for Cyclists

09:30 – 10:15 Session 1 Brainstorming

Challenges & Opportunities, Practical Usage, Successes & Failures of MR in AVs and Cyclists

10:15 – 10:30 Presentation & Discussion of the Results
10:30 – 10:45 Coffee Break
10:45 – 11:30 Session 2 Brainstorming

Elaborating concrete "user stories"
Translating these user stories into experimental designs

11:30 – 11:50 Presentation & Discussion of the Results
11:50 – 12:00 Wrap Up

Thanks for participating!

3 of 36

Agenda

Possibility to write a reflection statement/publication after the workshop

Results, outlook etc.

If interested, please email us:

andreas.riegler@thi.de

4 of 36

- Name�- Affiliation (University/Company)�- Do you have prior experience with � (automotive) AR/VR/MR?�- Interests/Expectations for this Workshop

Round of Introductions

5 of 36

- Automated Driving and Mixed Reality

- In-Vehicle Transitional Interfaces

- Multi-User VR

- AR for Cyclists

Introduction to the Topics

6 of 36

Motivation

Automated driving offers many advantages

Increased road safety^[1]
Improved traffic efficiency^[1]
Mobility for the impaired^[2]
Drivers can engage in non-driving related tasks:

viewing and writing emails, media consumption … ^[3]

Drivers consume an increasing amount of information while driving^[4]

In-car displays or smartphone

Extend vehicles into infotainment platforms and mobile offices

Automated Driving (AD), Automated Vehicles (AVs)

[1] B. van Arem, C. van Driel, R. Visser, „The Impact of Cooperative Adaptive Cruise Control on Traffic-Flow Characteristics”, IEEE Transactions on Intelligent Transportation Systems 7, 4, 2006.

[2] C. Harper, C. Hendrickson, S. Mangones, C. Samaras, „Estimating potential increases in travel with autonomous vehicles for the non-driving, elderly and people with travel-restrictive medical conditions”, Transportation Research Part C: Emerging Technologies 72, 2016.

[3] A. Riener, S. Boll, A. Kun, „Automotive User Interfaces in the Age of Automation”, Dagstuhl Seminar 16262, 2016.

[4] B. Pfleging, A. Schmidt, „(Non-) Driving-Related Activities in the Car: Defining Driver Activities for Manual and Automated Driving”, CHI’15, 2015.

[5] Volvo Car Group: Concept 26, delivering the luxury of time (PRNewsFoto/Volvo Car Group)

[5]

7 of 36

Automated Driving

SAE Levels of Vehicle Automation J3016 ^[1]

[1] SAE On-Road Automated Vehicle Standards Committee. 2021. Taxonomy and definitions for terms related to on-road motor vehicle automated driving systems.

8 of 36

Augmented Reality

A technology that superimposes a computer-generated image on a user’s view of the real world, thus providing a composite view.
“Terminator Vision” (1984)
Smartphone-based AR
Head-mounted Display (HMD)

Microsoft Hololens 2
MagicLeap 2

In-Vehicle: HUD/WSD

Introduction

9 of 36

Virtual Reality

Virtual reality (VR) provides a computer-generated 3D environment (including both computer graphics and 360-degree video)
VR surrounds a user and responds to an individual’s actions in a natural way
Why?

Immerse yourself in any computer-generated world

Room, City, Vehicle, Human Body, Universe

Explore unchartered territory of the human imagination
Immersive VR:

Perception of being physically present in a non-physical world
Often equipped with head-mounted display (HMD)

Applications:

Entertainment, Simulation, Training, etc.

Introduction

10 of 36

What is a Transitional Interface?

Reality-Virtuality (RV) Continuum by Milgram and Kishino

Introduction

11 of 36

In-Vehicle Transitional Interfaces

Establishment of in-vehicle mixed-reality user experiences
Novel user-centered methods for cross-virtuality information presentation and interaction need to be explored
Goal: Conception of cross-virtuality mechanisms to allow a seamless integration and transition between traditional 2D displays and 3D augmented reality (AR) windshield displays
Rapid prototyping of automotive human-machine interfaces (HMIs) can be achieved with cross-virtuality technology
Example: Virtual reality head-mounted displays (VR HMDs)

Introduction

12 of 36

In-Vehicle Transitional Interfaces

Conceptual design of AR interfaces with VR

Employ technologies across the RV continuum to sketch and prototypically develop automotive HMIs
Establish workflows on how transitional interface designers can utilize AR/VR technology for rapid prototyping

Handover and transition of visual content from 2D to 3D AR space

Introduce methods to move data visualizations displayed on a 2D (planar) display into 3D space for further usage in AR using a head-up or windshield display
Seamless handover, or switch, between devices (i.e., from smartphone to AR HUD)

Evaluation of in-vehicle transitional interfaces

Investigate methods and metrics on how to determine the usability and user experience of transitional interfaces using so-called complexity metrics

13 of 36

In-Vehicle Transitional Interfaces

Creating mockups for AR interfaces needs a different approach�than those in the mobile or desktop domain
We aim to utilize VR directly to rapidly sketch AR interfaces
Improved designer’s workspace by being immersed in the�virtual environment
Allows to prototype (in-vehicle) interfaces with various�parameters, such as content type, transparency, size,�rotation/tilt, distance …
Resulting interfaces will then be used for transitioning 2D�visualizations to the prototyped 3D AR interfaces

Conceptual Design of AR Interfaces with VR

14 of 36

In-Vehicle Transitional Interfaces

We need to determine how to handover visualization control from the planar 2D interface (e.g., smartphone, tablet) to the target AR interface
Ideally: seamless transition in order to emphasize the transformation of visual content and to highlight continuity
Convert head-down displays such as smartphones to head-up displays with immersive content presentations
Handover

Example: Swipe gesture

Use 2D display as controller

Handover and Transition of Visual Content from 2D to 3D AR Space

15 of 36

In-Vehicle Transitional Interfaces

Evaluate the usability and user experience of transitional�interfaces by the means of metrics
Performance on AR/VR tasks is important, but only part �of the picture

Visual Attention / Glance behavior
Motion sickness
Situational awareness
Eye & Neck Fatigue
Mental workload
Head movement
Balance and gait

Conduct a number of experimental studies to fully quantify�their impact on transitions from 2D displays into 3D space

Evaluation of In-Vehicle Transitional Interfaces

16 of 36

Multi-User VR

https://spectrum.ieee.org/forget-video-conferencinghost-your-next-meeting-in-vr

17 of 36

Multi-User VR

Shared experiences where users can communicate and interact in the same VR space

18 of 36

Multi-User VR

Shared experiences where users can communicate and interact in the same VR space

19 of 36

Multi-User VR

Shared experiences where users can communicate and interact in the same VR space

20 of 36