1 of 5

  Multimodal Emotion Recognition for �Human-Robot Interaction

1

PhD Student:                          Farshad Safavi

PhD Advisor:                            Dr. Ramana Vinjamuri 

2 of 5

2

Human Robot Interaction

  • Human-Robot Interaction (HRI) focuses on designing, evaluating, and understanding of robotic systems intended for interaction with humans (Cui et al., 2022). 
  • Robots can utilize diverse communication channels to interact with humans, such as hearing, speech, sight, touch, and learning.
  • Emotion recognition is essential for effective communication between humans and robots.

Physiological signals

Visual

Multiple Communities and Modalities

Haptics / Touch

Natural Language

Auditory

Background

Cui, Y., Song, X., Hu, Q., Li, Y., Sharma, P., & Khapre, S. (2022). Human-robot interaction in higher education for predicting student engagement. Computers and Electrical Engineering, 99, 107827. https://doi.org/10.1016/j.compeleceng.2022.107827

3 of 5

3

Human-Robot Collaboration

  • The block diagram illustrates the collaborative interaction between artificial systems and their users (Ciceri et al., 2008).
  • Emotion recognition by artificial systems enhances communication between robots and users, aiding in the achievement of shared objectives.
  • Joint action should be improved on the side of the artificial system by adapting emotionally to the human (Egger et al., 2019).

Emotional Attunement

Emotion Recognition

Robot

Artificial System

Joint action

Task

Actions

Human user

Goal

Non-verbal

Emotional

signals

Performance

Actions

(Ciceri et al., 2008)

Motivation

Ciceri, R. and S. Balzarotti, From signals to emotions: Applying emotion models to HM affective interactions., Affective Computing. InTech 3 (2008), p. 978

4 of 5

Multimodal Emotion Recognition for Human-Robot Interaction

  • We are developing a multimodal fusion model for a human-robot interaction system that controls a robotic arm through emotion recognition.
  • The main research focuses on how multimodal fusion can improve emotion recognition and also utilizes emotion recognition to directly influence the movements of robotic arms.
  • We developed a Transform-based model that detects emotion from facial expression.

​

Safavi, F., Patel, K., & Vinjamuri, R. K. (2023a). Towards Efficient Deep Learning Models for Facial Expression Recognition using Transformers. 2023 IEEE 19th International Conference on Body Sensor Networks (BSN), 1-4. https://doi.org/10.1109/BSN58485.2023.10331041

5 of 5

5

  • For the fusion approach, facial expression features and physiological features are concatenated.
  • The combined feature vector is then fed into a fusion classifier.
  • This classifier is a fully connected neural network.
  • The weights of the sub-classifiers are finalized and used to compute the fusion score.
  • We can experiment with different feature size combinations and assess the contribution of each individual modality, such as face appearance and bio-sensing.

Proposed Multimodal Fusion Model

Intermediate Fusion

face feature vector

EEG feature vector

fusion feature

classifier