20: Application Areas - Part 1

Juho Kim & Jean Young Song

Human-AI Interaction KAIST Fall 2020 | kixlab.org/courses/human-ai

Administrative Notes

• Project Final Presentation is 6/10
• Project Final Report and Video is due 6/22 Tue after the final week

For final presentation, you'll have 10 minutes to do the following:

• Motivate the problem.
• List the main tasks you're trying to support.
• What is your solution? How is it different from existing systems? Why is it better?
• Walk through your interface. You can use video or screenshots, or even do a live demo. But be careful with live demos, because they require a lot of practice and include many moving parts.
• Show us how your deployment went. What are some exciting results that suggest your application served the tasks well? What did you learn from designing human-AI interactions?
• Limitations, implications, and possible improvements.

After the presentation, you'll have 3 minutes for Q and A.

​

Previously on CS492E...

Human-AI Collaboration

Scenario: Moving a heavy sofa to the door

Images: https://robots.ieee.org/robots/hubo/hubo-thumb@2x.jpg

https://www.accessselfstorage.com/wp-content/uploads/2018/06/furniture.png

Imagine a human-robot team moving a heavy sofa together to the door. How should the following be supported �for successful teamwork?

​

• Individual plan as part of a joint plan
• Commitment to partner’s action
• Commitment to mutual belief
• Commitment to mutual support
• Grounding
• Joint closure

Previously on CS492E...

Human-AI Interaction in AXIS

Human

Better quality explanation

Better learning

AI

Solves problems

Provides explanations

Rates explanations

Updates policy using bandits + Thompson sampling

Interaction

Reflection on

in-class activity

Today’s Learning Objectives

After today’s class, you should be able to...

• Be familiar with Accessibility & HAI
• Be familiar with Robotics & HAI
• Think about other interesting human-AI interaction applications.

​

Accessibility

Accessibility can be anyone’s problem

• > 15% of the U.S. population has a disability
• You have or will have a disability (when you get older)
• Situational impairments affect us all

​

Stats from Prof. Jeff Bigham’s Stanford Seminar talk titled

“Accessibility and the AI autumn” https://www.youtube.com/watch?v=PJE_gnTreBo

​

Why is it more problematic these days?

The information gap between disabled and abled people might be larger these days.

Bigham, Jeffrey P., Ryan S. Kaminsky, Richard E. Ladner, Oscar M. Danielsson, and Gordon L. Hempton. "WebInSight: making web images accessible." In Proceedings of the 8th international ACM SIGACCESS conference on Computers and accessibility, pp. 181-188. 2006.

Why is it more problematic these days?

​

​

​

​

​

​

How the computer sees us — Dan O’Sullivan & Tom Igoe’s model, 2004. The image illustrates how imbalanced is the use of individual body parts in man-computer interaction.

https://nexocode.com/blog/posts/gestures-recognition-challenges-and-opportunities/

What can we do to making images accessible?

Ask users to write image descriptions called Alt Text.

“When composing image descriptions, it may seem difficult to decide what to include. Here are features that can be included in image descriptions:

• Placement of objects in image
• Image style (painting, graph)
• Colors
• Names of people
• Clothes (if they are an important detail)
• Animals
• Placement of text
• Emotions, such as smiling
• Surroundings.

​

https://www.perkinselearning.org/technology/blog/how-write-alt-text-and-image-descriptions-visually-impaired

What can we do to making images accessible?

Ask users to write image descriptions called Alt Text.

“When composing image descriptions, it may seem difficult to decide what to include. Here are features that can be included in image descriptions:

• Placement of objects in image
• Image style (painting, graph)
• Colors
• Names of people
• Clothes (if they are an important detail)
• Animals
• Placement of text
• Emotions, such as smiling
• Surroundings.

​

It’s not really that easy for users to create!

https://www.perkinselearning.org/technology/blog/how-write-alt-text-and-image-descriptions-visually-impaired

Making images accessible

How many of you have written alt text when posting images on social network?

Image link

Human-AI Interaction to Improve Accessibility

How would AI help improve accessibility?

Can you think of any examples?

Accessibility + Device + AI

Robotics

Topics in Human-Robot Interaction

• Social interaction (Emotional support, Greeting, etc)
• Robot companion
• Appearance
• Personalization
• Collaborative task completion
• Independent task completion
• Optimization
• Inverse RL
• etc

Robots need to understand human behavior to interact with humans

​

Examples:

• Companion robot providing emotional support to older adults
• Autonomous vehicles driving with other cars and pedestrians
• Service robots guiding blind people
• etc

Robots need to understand human behavior to interact with humans

Robots need to understand human behavior to interact with humans

​

Challenge: However, modeling human behavior is difficult! Human behavior is stochastic and noisy, many times even irrational.

“Furthermore, when GRACE made an attempt to start an interaction, participants were more likely to respond positively (by turning to the robot, looking at it, or touching its screen) in the reception and banquet. In the hallway, on the other hand, people were equally likely to engage as they were to disengage from the interaction in response to GRACE’s interactive behavior”

Warm-up: a quick in-class activity

​

Accessibility + Robotics + AI

​

Question: What possible applications are there in the intersection of accessibility + robot + AI?

Reading Responses

Software Engineering for Machine Learning: A Case Study (ICSE-SEIP 2019.)

​

“As a key finding of our analyses, we discovered three fundamental differences to building applications and platforms for training and fielding machine-learning models than we have seen in prior application domains.”

​

​

​

​

If you are engineering a robot to interact with human based on AI technique, what would you change in the above workflow?

ACTIVITY: Accessibility, Robotics, and AI

• Task 1: Come up with a real/concrete accessibility + AI application example. You can search the web or YouTube.
• Task 2: Discuss unique challenges in accessibility + AI applications compared to other HAI interaction applications we discussed in the course.
• Task 3: Come up with a real/concrete AI-powered robot interaction application example. You can search the web or YouTube.
• Task 4: Discuss unique challenges in AI-powered robot interaction applications compared to other HAI interaction applications we discussed in the course.
• Teams of 3, 20 mins

www.yellkey.com/special

​

​

​