Emotion aware conversational interface

πŸ˜¬πŸ’–

Hi my project is emotion aware conversational interface.

And you might noticed my first slide represent my emotions. I am kinds of nervous, but happy to have you guys.

About Me

Hong Min

<Research Interest>

User experience design, Deep learning(CNN, RNN, GAN), NLP, CV.

<Project & education>

AI-Art lab, generate Korean poetry,

, Research internship at RIST.

(2018 03-04)

Self-driving RC car by a end to end learning

, education program at RIST

(2018 01-02)

https://gitlab.com/minh364/FreeRun

https://github.com/yisu-kim/FreeRun

B.S. Fine art. at Seoul Nat’l Univ S&T.

Mentor : Seok Jin Hong @ SKT

mimmh93@gmail.com

+82)010-2225-6474

My name is hong min. And my mentor is seok jin hong from SKT. he gave a lot of advice. And also other mentors like eric, νƒœλ¦¬, ꡬ쀀, sourup and more shared thier knowledge and gave lot of help.

Thank you for inviting me this awesome platform.

My backgroud is here. As research interest item saids, I also interested in User experience design. So I approach my project in application level.

Color = emotion

My main idea is that mapping colors to emotions

My main idea is that mapping colors to emotions

Today was disaster...

I’m sorry to hear that..

So what?

This messenger could contains more face-to-face conversational information.

It could be applying in messenger for chatting, which recognizes the user's emotion and displays it in colors.

It could be applying in messenger for chatting, which recognizes the user's emotion and displays it in colors. (and emoji)

Proposed method

<apply to messenger>

text

emotion

classifier

Emotion label

Mapping to rgba

COLOR CODES

For do that. I classifying text to emojis and mapping colors to emotion labels.

For do that. I classifying text to emojis and mapping colors to emotion labels.

Live Demo

I make a demo web page that briefly covered this concept.

And I prepare little game.

I will present little gift to the first person who finds

I make a demo web page that briefly covered this concept.

And I prepare little game.

I will present little gift to the first person who finds

Fines this color fastest !

/*I found that the display color of Android devices and my computer are very different.

so If you find a similar color, I will present the gift, please raise your hand.*/

THIS COLOR

Its been a min,

Hint !

πŸ˜•πŸ˜žπŸ˜·

πŸ’πŸ˜πŸ˜Œ

Fines this color fastest !

I got this color with this sentence. β€œI am so fine”

Fines this color fastest !

/*I found that the display color of Android devices and my computer are very different.

so If you find a similar color, I will present the item, please raise your hand.*/

THIS COLOR

Hint !

πŸ˜ŽπŸ˜ŒπŸ‘Œ

THIS COLOR

Fines this color fastest !

I got this color with this sentence. β€œI am good”

I got this color with this sentence. β€œI am good”

Fines this color fastest !

/*I found that the display color of Android devices and my computer are very different.

so If you find a similar color, I will present the gift, please raise your hand.*/

Hint !

πŸ˜•πŸ˜žπŸ˜·

Fines this color fastest !

I got this color with this sentence. β€œnot really good”

Fines this color fastest !

/*I found that the display color of Android devices and my computer are very different.

so If you find a similar color, I will present the gift, please raise your hand.*/

Hint !

πŸ˜”πŸ˜žπŸ˜’

Fines this color fastest !

I got this color with this sentence. β€œI am sorry”

Here are sentences from audience about p17-19

...

β€œI feel blue”
...

model - DeepMoji

T = text length

C = the number of classes

I use the DeepMoji model from MIT media lab as emotion classifier. It is trained by 1246 million tweets, which is containing one of 64 different common emoticon. There are embedding layer to project each word into a vector space. ( a hyper tangent activation enfoce a constraint of each embedding dimension being within [-1,1] two bidirectional LSTM layers to capture the context of each word. And an attention layer that lets the model decide the importance of each word for the prediction.

Color mapping to Emoji Dendrogram

(to rgb)

I mapping color based on dendrogram. The dendrogram shows how the model learns to group emojis based on emotional content. The y-axis is the distance on the correlation matrix of the model’s predictions.

It measured using average linkage.

Color mapping (rgba)

alpha : defines the opacity as a number

0.0 fully transparent

1.0 fully opaque

~

The color code I use is rgba.

top 1

😷

top 2

😌

top 3

πŸ‘Œ

Emoji probability array

( πŸ˜‚, πŸ˜’, 😩, 😭, '😍', 'πŸ˜”', 'πŸ‘Œ', '😊', '❀', '😏',

😁, 🎢, 😳', 'πŸ’―, '😴', '😌', '☺', 'πŸ™Œ', 'πŸ’•', 'πŸ˜‘',

πŸ˜…, πŸ™, 'πŸ˜•', '😘, 'β™₯', '😐', 'πŸ’', '😞', 'πŸ™ˆ', '😫',

✌, '😎', '😑', 'πŸ‘, '😒', 'πŸ˜ͺ', 'πŸ˜‹', '😀', 'βœ‹', '😷',

πŸ‘, πŸ‘€', 'πŸ”«', '😣, '😈', 'πŸ˜“', 'πŸ’”', 'πŸ’“', '🎧', 'πŸ™Š',

πŸ˜‰, 'πŸ’€', 'πŸ˜–', 'πŸ˜„, '😜', '😠', 'πŸ™…', 'πŸ’ͺ', 'πŸ‘Š', 'πŸ’œ',

πŸ’–, 'πŸ’™', '😬', '✨, float32)

norm = top1 + top2 + top3

Each layer’s opacity =
each probability of top1 to top3 / norm

The output from the model is the probability of each 64 different emojis.

I use top 3 probability with normalization for define the opacity of the layer.

And these 3 layers are overlapped.

And determine the color of the screen.

In this way I represent the emotion by color.

Proposed method

<apply for stage>

voice

Text: real-time subtitle

emotion

classifier

Emotion label

Mapping to

COLOR CODES

Speech

recognition

: Stage effect

which interacts with actors acting

This concept also can be applying on stage effect, which interacts with actors acting in real time.

Proposed method

<apply to chat bot>

voice

text

: response that emotionally controlled

emotion

classifier

chat bot

Emotion label

Mapping to

COLOR CODES

Speech

recognition

Input = <special token EMO> + sentence

I don’t want talking with you.

:screen color

It could be apply on chat bot. If background color is changed by user and bot’s messege, than user can regonize thier emotion in color. Than thay would feel they are sharing emotion with chat bot.

And when train chat bot, add special token <emotion label> on input, than chat bot might be learn emotional context. I want to working on this on the near future.

Thank you!

πŸ™πŸ™ŒπŸ‘

hong - Google Slides