A CROSS-PLATFORM AI APPLICATION FOR HUMAN-ROBOT INTERACTION

Nicolò, Mantovani

Stefano, Ditrani

OUTLINE

1. Introduction
2. Background and tools
3. Architecture
4. Demo
5. Conclusion

HOW MUCH DO YOU REALLY KNOW ABOUT THE WORLD YOU LIVE IN?

HTTPS://WWW.YOUTUBE.COM/WATCH?V=9YFL83L3D8U�

VIDEO

INDUSTRIAL VS COLLABORATIVE ROBOTS�

1953 First Teleoperation

experiments

1961 Fondation of Unimation and puma Robot

‘90 New architectures of robots

2000 New generation of Robots

‘80 Robots used in Production and Automation industrial

WHAT IS A ROBOT?

A robot is a machine

which can sense, plan and act

Artificial Intelligence

Actuators

Sensors

Are appliances considered robots?

FROM PERSONAL COMPUTERS TO PERSONAL ROBOTS

Human Robot Interaction

​

How Do Robots Interact With Humans?

COGNITIVE HUMAN ROBOT INTERACTION

National Museum Of Korea – Robot Guide

Amazon Robot

PHYSICAL HUMAN ROBOT INTERACTION

WEARABLE AND TELEROBOTICS

Field dedicated to design robotic systems for use by or with humans.

HRI

NOWADAYS

Working together to accomplish a common goal.

From industrial to social robots (healthcare system, education, entertainment and commercial purposes, etc.).

​

ROBOTICS MARKET

Source: International Federation of Robotics

$13.3 ml

WHO WE ARE

​

​

​

​

​

​

Integrating multiple AI components for Human-Robot Interaction tasks

ROS-BASED PLATFORM: designed for robot using

ROS as communication layer

MODULAR & FLEXIBLE: The system is designed to run on any machine having Docker installed

An Open Teleconference Toolkit for Robotics

MULTI-PLATFORM: designed according to the

model-View-Controller Pattern

OPEN & EXTENSIBLE: The system is designed to be extensible to new voice inputs and to any new services offered by the robot it operates with

NEW GOAL

​

​

​

​

​

​

A flexible and modular robot architecture platform for Human Robot Interaction applications to easily support multiple robot platforms.

​

​

Robotics and AI components all together.

​

​

BACKGROUND AND TOOLS

TECHNOLOGIES

STRONG DEPENDENCIES BETWEEN

SOFTWARE COMPONENTS

CONTAINERIZATION

​

​

​

​

​

​

CONSISTENT AND ISOLATED ENVIRONMENT

ROBOT OPERATING SYSTEM (ROS)

COMPUTATION GRAPH LEVEL

COMMUNICATION IN ROS

OPEN-SOURCE FRAMEWORK FOR DEVELOPING

AND

USING ROBOTIC SOFTWARE

​

​

​

​

​

​

Nodes: running processes meant to be executable code (single device or distributed)

Topics: named buses over which nodes exchanges data (many-to-many publisher-subscriber pattern)

Messages: data structure exchanged by nodes (customized or primitives)

Master: core of the network and nodes tracker

​

​

ROBOT ARCHITECTURE

DOCKER ARCHITECTURE

AI layer �AI modules responsible for processing incoming images from the robot camera (according to the different pipelines)

​

Robot layer �Core of the robot functionalities (i.e., interfaces that communicate with sensors and actuators)

​

Plan-actions layer�It contains actions and fluents implementation as well as the framework for executing and monitoring plans

​

DOCKER ARCHITECTURE

AI layer �AI modules responsible for processing incoming images from the robot camera (according to the different pipelines)

​

Robot layer �Core of the robot functionalities (i.e., interfaces that communicate with sensors and actuators)

​

Plan-actions layer�It contains actions and fluents implementation as well as the framework for executing and monitoring plans

​

ROBOT LAYER

DOCKER ARCHITECTURE

AI layer �AI modules responsible for processing incoming images from the robot camera (according to the different pipelines)

​

Robot layer �Core of the robot functionalities (i.e., interfaces that communicate with sensors and actuators)

​

Plan-actions layer�It contains actions and fluents implementation as well as the framework for executing and monitoring plans

​

AI LAYER – PERSON DETECTION

• Base network (VGG-16 based)�strong performance in high quality image classification�
• Auxiliary network�Layers that decrease in size progressively and allow predictions of detections at mutliple scales

​

• Weighted loss

​

• Fixed priors and IoU (intersection over union)

​

• Technologies

SINGLE SHOT MULTIBOX DETECTOR (SSD)

AI LAYER (CONT’D) – FACE DETECTION

CASCADE CLASSIFIER

DOCKER ARCHITECTURE

AI layer �AI modules responsible for processing incoming images from the robot camera (according to the different pipelines)

​

Robot layer �Core of the robot functionalities (i.e., interfaces that communicate with sensors and actuators)

​

Plan-actions layer�It contains actions and fluents implementation as well as the framework for executing and monitoring plans

​

CLASSICAL

AI APPROACH

​

PROBLEM SOLVING

PLANS, ACTIONS AND FLUENTS

• Problem to solve

​

• Language: STRIPS, SAS+, PDDL (de facto standard)

​

• Planner: algorithm used for solving the search, both informed (BFS, A*) and uninformed (BDS, DFS, IDFS, etc.)

​

• Solution: plan

PLANS, ACTIONS AND FLUENTS

PLAN�A sequence of actions that applied to the initial state led to the goal

EXAMPLE

What: AI agent welcomes people detected

Where: in a museum

�

1. Sense people;
2. Welcome people;

PLAN-ACTIONS LAYER - PNP

1 goto_printer1;

2 sense_personhere;

2 < personhere ?

3. say_hello :

4 (not personhere) ?

5 >;

6 goto_home;

ROS String message to start plan

1

2

3

PLAN-ACTIONS LAYER - ACTIONS

Increased interoperability between a Plan Execution Monitor (PEM) and actions/fluents�

Actions

• Start (PEM to PLEXI)
• Interrupt (PEM to PLEXI)
• Resume (PEM to PLEXI)
• End (PLEXI to PEM)

PLEXI (wrapper/proxy)

PLAN-ACTIONS LAYER - FLUENTS

Fluents:

• getValue (PEM to PLEXI)
• setValue (PLEXI to PEM)

FLOW EXECUTION

FLOW EXECUTION

PNP

PEM reads fluent value and resume plan execution

Plan end

ACTIONS

FLOW EXECUTION

PNP

PEM reads fluent value and resume plan execution

Plan end

ACTIONS

FLOW EXECUTION

PNP

PEM reads fluent value and resume plan execution

Plan end

ACTIONS

FLOW EXECUTION

PNP

PEM reads fluent value and resume plan execution

Plan end

ACTIONS

FLOW EXECUTION

PNP

PEM reads fluent value and resume plan execution

Plan end

ACTIONS

FLOW EXECUTION

PNP

PEM reads fluent value and resume plan execution

Plan end

ACTIONS

FLOW EXECUTION

PNP

PEM reads fluent value and resume plan execution

Plan end

ACTIONS

FLOW EXECUTION

PNP

PEM reads fluent value and resume plan execution

Plan end

ACTIONS

FLOW EXECUTION

PNP

PEM reads fluent value and resume plan execution

Plan end

ACTIONS

DEMO

GUEST PRESENCE: OWNER BUSY STATUS

DEMO

…

1 sense_mypersonhere;

2 < mypersonhere ?

welcome :

4 (not mypersonhere) ?

5 >;

1 waitforperson;

2 say_busy;

ROS_MASTER_URI (to MARRtino)

ROS_IP (to DHCP subnet laptop IP)

​

NAO_URI

ROS-BASED

NAO-BASED

DEMO

• My laptop
• Petri Net Plans (PNP)
• Actions

​

• MARRtino (robot)
• Robot
• Vision
• Navigation
• Speech

​

• Third part
• Person Detection
• Face Detection
• Face Expression Recognition

​

Robust conditional plan

Changes:�

ROS_MASTER_URI (to MARRtino)

ROS_IP (to the DHCP subnet laptop IP)

1 sense_mypersonhere;

2 < mypersonhere ?

3. welcome :

4 (not mypersonhere) ?

5 >;

ONGOING WORK

CONCLUSION

• Extend the current architecture to cross-platform
• Implement new use cases for NAO (new tasks)
• Integrating new AI modules
• Update current AI components with state-of-the-art model

​

THANK YOU

www.reply.com