1 of 27

COACHES�COoperative Autonomous Robots in Complex and Human EnvironmentS

Coordinator : Abdel-Illah Mouaddib

Partners : University of Caen Basse-Normandie, France

University of Sapienza Roma, Italy

Vrije Universiteit Brussel , Belgium

Sabanci University , Istanbul, Turkey

Participants: Agglomeration Caen la mer, �Ville de Caen, Rive de L’orne Mall, Caen

SME Algorithmica, Italy

​

1

http://coaches.greyc.fr

2 of 27

People

  • University of Caen Basse-Normandie, France - Abdel-Illah Mouaddib

​

  • Sabanci University , Istanbul, Turkey - Esra Erdem

​

  • Vrije Universiteit Brussel , Belgium - Hichem Sahli

​

  • University of Sapienza Roma, Italy
    • Luca Iocchi PI
    • Daniele Nardi co-PI
    • Giorgio Grisetti
    • Maria Teresa Lazaro
    • Domenico Bloisi
    • (students)

​

2

2

3 of 27

Motivation

  • Deploying robots in public areas to assist, guide, and escort visitors and surveillance
  • Guidance and Personal assistance in public areas:�shop malls, touristic sites, airport, train stations, hospitals
  • Surveillance and patrolling in cooperative way with intervention units (firefighters, police, medical units, and tourist office staff, …)
  • Our target use cases:
    • Caen Rive de l’Orne Shopping Mall in cooperation with Caen city

3

3

4 of 27

Scientific challenges

  • Environment monitoring
    • Environment modeling
      • KB event language, 2D/3D visual representation of the environment
    • Rule-based semantic scene analysis
      • Detecting object, people behaviors and tracking
  • Scene understanding and situation awareness
    • Human activities Recognition
      • Human needs and activity representation
    • Situation awareness
      • Detecting behavior and abnormal events
  • Multi-modal short-term human-robot interaction
    • From robot to human
      • Guiding task : speech, arm pointing, moving to destination, Information assistance task: text and maps on the screen
    • From human to robot
      • Requests of guidance or information: text, speech
  • Multi-Robots cooperative planning
    • Goal-Based Interaction Decentralized Decision-Making for Multi-agent Planning
    • Robust and safe Navigation in populated areas

4

4

5 of 27

Software and data flow

5

5

6 of 27

Software and data flow

6

6

7 of 27

Key challenges

Advance state of the art of autonomous systems acting in public environments and interacting with naive users�

    • Increase the robustness of multi-robot systems
    • Increase the autonomy of robotic systems�
      • situation awareness
      • navigate safely in crowded environments
      • distributed planning and decision making
      • short-term interactions with naive users

​

7

7

8 of 27

Key results

  • Ability to
    • navigate in a crowded environment
    • detect people
    • interact with them
    • assist them
    • distributed planning
  • Deployment in
    • shopping mall
    • center of congress
    • hospital
    • IAB Forum

​

8

8

9 of 27

Partners

  • Caen Normandie, France
    • KB reasoning
    • Multi-Robots Task Planning
    • Decision-Making
  • Sapienza, Rome, Italy
    • Human-Robot Interaction
    • Robust navigation
    • Decision-Making
  • Sebanci, Istanbul, Turkey
    • Multi-KB reasoning
  • Vreij, Brussels, Belgium
    • Image and video processing

​

9

9

10 of 27

Sapienza goals

​

  • A0.1 Gestione del progetto
  • A0.2 Disseminazione dei risultati
  • A1.1 Modellazione dell’ambiente basato su conoscenza
  • A3.1 Interazione uomo-robot multi-modale
  • A4.1 Navigazione sicura e robusta in ambiente popolato da persone
  • A4.2 Tecniche decisionali decentralizzate per sistemi multi-robot cooperativi
  • A5.1 Definizione dell’architettura interna
  • A5.2 Definizione dei casi d’uso
  • A5.3 Set-up e messa in opera dei robot mobili e dell’infrastruttura di comunicazione
  • A5.4 Realizzazione delle dimostrazioni
  • A5.5 Valutazione delle performance dell’intero sistema

​

​

10

11 of 27

Sapienza contributions

11

Decision Making Architecture

​

Multi-Modal Interaction

​

Robust and safe navigation

​

HRI & Planning

​

System integration & demos

​

https://sites.google.com/a/dis.uniroma1.it/�coaches-sapienza

A5.1 … A5.5

A1.1

A4.2

A3.1

A4.1

12 of 27

Decision Making

12

12

L. Jeanpierre, A.-I. Mouaddib, L. Iocchi, M.T. Lazaro, A. Pennisi, H. Sahli, E. Erdem, E. Demirel, V. Patoglu.

COACHES: An assistance multi-robot system in public areas.

In Proc. European Conference on Mobile Robots (ECMR), 2017.

A5.1

A1.1

13 of 27

Multi-Modal Interaction

​

  • Multi-modal short-term human-robot Interaction
    • Developing a simple language for defining interaction template
  • Human intention estimation
    • Estimating the behavior through the moving
    • Dialogue

​

13

13

ask<whichActivity, {shop, restuarant}>

ask<ComeFrom, “country”, …>

state <ShowInfo, Country>

Paola Ferrarelli, María T. Lázaro and Luca Iocchi.

Design of robot teaching assistants through multi-modal human-robot interactions.

In Proc. of International Conference on Robotics in Education (RiE) 2017��L. Iocchi, M. T. Lazaro, L. Jeanpierre, A.-I. Mouaddib

Personalized short-term multi-modal interaction for social robots assisting users in shopping malls

International Conference on Social Robotics (ICSR), Paris, France, 2015

​

A3.1

14 of 27

Safe Navigation

​

Navigation in dynamic environments

​

  • Self-Localization
  • Path Planning
  • Obstacle avoidance
  • Mapping

​

Robustness, safety

14

14

A4.1

Maria Teresa Lázaro, Giorgio Grisetti, Luca Iocchi, Jaime Pulido Fentanes, Marc Hanheide: A Lightweight Navigation System for Mobile Robots. ROBOT (2) 2017: 295-306

​

​

​

15 of 27

HRI & Planning

  • Planning for HRI
  • MDP Planning and execution
  • On-line negotiations
  • Conditional planning
  • Human-aware planning
  • Social rules / social plans

​

15

V. Sanelli, M. Cashmore, D. Magazzeni, and L. Iocchi. Short-Term Human Robot Interaction through Conditional Planning and Execution. In Proc. of International Conference on Automated Planning and Scheduling (ICAPS) 2017.�

E. Sebastiani, R. Lallement, R. Alami, and L. Iocchi. Dealing with On-line Human-Robot Negotiations in Hierarchical Agent-based Task Planner. In Proc. of International Conference on Automated Planning and Scheduling (ICAPS) 2017.��L. Iocchi, L. Jeanpierre, M. T. Lazaro, A.-I. Mouaddib.

A practical framework for robust decision-theoretic planning and execution for service robots. Proc. of International Conference on Automated Planning and Scheduling (ICAPS) 2016.

A4.2

F. M. Carlucci, L. Nardi, L. Iocchi, D. Nardi. Explicit Representation of Social Norms for Social Robots. Proc. of International Conference on Intelligent Robots and Systems (IROS), 2015

�L. Nardi, L. Iocchi. Representation and Execution of Social Plans through Human-Robot Collaboration. Proc. of Fifth International Conference on Social Robotics (ICSR) 2014.

​

​

16 of 27

Robot Prototype

DIAGO

  • Segway mobile base
  • Laser range finder
  • 2 RGBD cameras
  • Microphone
  • Speakers
  • Laptop (Intel i7, 8 GB RAM, GPU)
  • ROS drivers
  • ROS basic functionalities
  • ROS COACHES packages

​

16

A5.3

17 of 27

Robot Design

17

A5.3

18 of 27

Robot Development

18

A5.3

19 of 27

System Integration & Demos

19

19

A5.4

A5.5

20 of 27

Publications

20

20

2014

​

L. Nardi, L. Iocchi.

Representation and Execution of Social Plans through Human-Robot Collaboration.

Fifth International Conference on Social Robotics (ICSR) 266-275, 2014.

​

2015

​

Fabio Ferrari, A.-I Mouaddib.

Hierarchical POMDP for Escort Task.

Arlington, AAAI Fall Symposium on Sequential Decision Making for Intelligent Agents, November 2015.

L. Iocchi, M. T. Lazaro, L. Jeanpierre, A.-I. Mouaddib.

Personalized short-term multi-modal interaction for social robots assisting users in shopping malls.

International Conference on Social Robotics (ICSR), Paris, France, Oct. 26-30, 2015.

F. M. Carlucci, L. Nardi, L. Iocchi, D. Nardi.

Explicit Representation of Social Norms for Social Robots.

International Conference on Intelligent Robots and Systems (IROS), Hamburg, Germany, Sept. 28-Oct. 2, 2015.

2015

​

L. Iocchi, M. T. Lazaro, E. Erdem, L. Jeanpierre, A.-I. Mouaddib, H. Sahli.

COACHES: Cooperative Autonomous Robots in Complex and Human Populated Environments.

14th Conference of the Italian Association for Artificial Intelligence, Ferrara , Sept. 23-25, 2015.

R. Capobianco, J. Serafin, J. Dichtl, G. Grisetti, L. Iocchi, D. Nardi. A Proposal for Semantic Map Representation and Evaluation. In Proc. of the European Conference on Mobile Robots (ECMR), Lincoln, United Kingdom, 2015.

Andrea Pennisi, Fabio Previtali, Domenico D. Bloisi and Luca Iocchi. Real-Time Adaptive Background Modeling in Fast Changing Conditions. In Proceedings of 12th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 2015.

A0.2

21 of 27

Publications

21

21

2017

​

Valerio Sanelli, Michael Cashmore, Daniele Magazzeni, and Luca Iocchi.

Short-Term Human Robot Interaction through Conditional Planning and Execution. In Proc. of International Conference on Automated Planning and Scheduling (ICAPS) 2017.

​

Eugenio Sebastiani, Raphaël Lallement, Rachid Alami, and Luca Iocchi. Dealing with On-line Human-Robot Negotiations in Hierarchical Agent-based Task Planner. In Proc. of International Conference on Automated Planning and Scheduling (ICAPS) 2017.

​

Paola Ferrarelli, María T. Lázaro and Luca Iocchi.

Design of robot teaching assistants through multi-modal human-robot interactions.

In Proc. of International Conference on Robotics in Education (RiE) 2017

​

Fabio-Valerio Ferrari, Laurent Jeanpierre and Abdel-Illah Mouaddib, ​​Flexible POMDP Framework for Human-Robot Cooperation in Escort Tasks, International Joint Conference on Autonomous Agents and MultiAgent Systems (AAMAS), extended abstract, Sao Palo, Brazil, May, 2017.

​

2016

​

L. Iocchi, L. Jeanpierre, M. T. Lazaro, A.-I. Mouaddib.

A practical framework for robust decision-theoretic planning and execution for service robots. International Conference on Automated Planning and Scheduling (ICAPS), London, UK, June 12-17, 2016.

​

Esra Erdem, Michael Gelfond, Nicola Leone:

Applications of Answer Set Programming. AI Magazine 37(3): 53-68 (2016)

​

Esra Erdem, Volkan Patoglu, Peter Schüller:

A systematic analysis of levels of integration between high-level task planning and low-level feasibility checks. AI Commun. 29(2): 319-349 (2016)

​

Ezgi Demirel, Kamil Doruk Gur, Esra Erdem:

Human-Robot Interaction in a Shopping Mall: A CNL Approach. CNL 2016: 111-122

​

Esra Erdem, Volkan Patoglu, Zeynep Gozen Saribatur:

Diagnostic Reasoning for Robotics Using Action Languages. LPNMR 2015: 317-331

​

​

​

A0.2

22 of 27

Software release

​

22

22

A3.1

A4.1

A4.2

23 of 27

Dissemination

  • Polyclinique saint-martin (private hospital), Caen: a demonstration to assist people to find a doctor or a unit care and new capabilities of the robot are under discussion like transporting goods, medicines, …
  • Relais science, science, culture and innovation, Caen: a partnership between relais sciences and our group to develop demonstration at their modern and high tech building with many and reconfigurable facilities. A demonstration of smart home and robots is under discussion.
  • Luca Iocchi  “Conditional planning for Human-Robot Interaction”, PlanRob Dagstuhl Seminar 15-20 gennaio, 2017, Dagstuhl , Germany.
  • Luca Iocchi  "Short-Term Human Robot Interaction through Conditional Planning and Execution", ICAPS 2017
  • Luca Iocchi  "Dealing with On-line Human-Robot Negotiations in Hierarchical Agent-based Task Planner", ICAPS 2017
  • Abdel-Illah Mouaddib, Smart City event organized by Caen City, “ Robots in public space”, 2017
  • Abdel-Illah-Mouaddib and Laurent Jeanpierre, Google French Tour, December 2017, Caen
  • Coaches.greyc.fr the web site of the project is maintained and up to date with new results and demonstration (pictures, video, publications, …).

​

23

23

A0.2

24 of 27

Dissemination

24

24

A0.2

25 of 27

Follow-up

  • AAAI Fall Symposia “Artificial Intelligence for Human-Robot Interaction” (AI-HRI), Arlington, Virginia USA, 2017 - 2019�
  • ICAPS 2017 Tutorial on AI Planning for Robotics and Human-Robot Interaction, Pittsburgh, USA, June 19, 2017�
  • ERF 2017 Workshop on HRI in public spaces with naive users - Edimburgh, UK, - March 22, 2017�
  • RoboCup@Home SPQReL team� - RoboCup 2017, Nagoya, Japan - July 27-31, 2017� - RoboCup 2018, Montreal, Canada - June 16-22, 2018

​

25

25

26 of 27

Conclusions

​

  • A multi-service multi-robot system and programmable services
  • A software for service multi-robot and independent from the physical platforms
  • Possible deployments in different environments

​

26

26

27 of 27

Thank you

27

27